Skip to content

v0.3.0

Release v0.3.0: Scatter, Signal Gate, and Rate Limiting
@nshkrdotcom nshkrdotcom tagged this 19 Dec 01:11 
This release introduces three major features that extend FlowStone's
orchestration capabilities for complex distributed workloads.

Scatter (Dynamic Fan-Out)
-------------------------
Scatter enables runtime-discovered parallel execution within FlowStone's
asset-centric model. An asset can fan out into N instances based on data
from upstream dependencies, execute in parallel with configurable
concurrency controls, and reconverge for downstream consumption.

Key components:
  - FlowStone.Scatter: Core coordination logic with barrier-based tracking
  - FlowStone.Scatter.Barrier: Ecto schema tracking scatter completion
  - FlowStone.Scatter.Key: Deterministic key serialization and hashing
  - FlowStone.Scatter.Options: Configurable execution parameters
  - FlowStone.Scatter.Result: Per-instance result storage with compression
  - FlowStone.Workers.ScatterWorker: Oban worker for parallel execution

DSL additions: scatter, scatter_options, gather, max_concurrent, rate_limit,
failure_threshold, failure_mode, retry_strategy, max_attempts, queue,
priority, timeout

Scatter supports partial failure modes with configurable thresholds,
allowing pipelines to continue when a subset of instances fail below the
specified tolerance level.

Signal Gate (Durable External Suspension)
-----------------------------------------
Signal Gate enables FlowStone assets to durably suspend execution while
waiting for external signals such as webhooks, callbacks, or human
approvals. Unlike polling approaches, Signal Gate consumes zero resources
while waiting and provides immediate resumption upon signal receipt.

Key components:
  - FlowStone.SignalGate: Core gate management with secure token validation
  - FlowStone.SignalGate.Gate: Ecto schema for suspension state
  - FlowStone.SignalGate.Token: HMAC-signed token generation and validation
  - FlowStone.Workers.SignalGateTimeoutWorker: Per-gate timeout handling
  - FlowStone.Workers.SignalGateSweeper: Safety net for missed timeouts

DSL additions: on_signal, on_timeout

Tokens are cryptographically signed with expiration timestamps, preventing
forgery and enabling secure webhook integrations. Timeout handling supports
retry with exponential backoff or immediate failure modes.

Global Rate Limiting
--------------------
FlowStone.RateLimiter provides distributed rate limiting using Hammer for
token bucket rate limiting and Postgres advisory locks for semaphore-based
concurrency control.

Capabilities:
  - Per-bucket rate limiting with check/2 and with_limit/4
  - Semaphore slots with acquire_slot/2, release_slot/2, with_slot/4
  - Status inspection and bucket reset
  - Integration with Scatter for per-barrier concurrency limits

Observability
-------------
New telemetry events for comprehensive monitoring:
  - Scatter: start, complete, failed, cancel, instance_complete, instance_fail,
    gather_ready
  - Signal Gate: create, signal, timeout, timeout_retry, cancel
  - Rate Limiting: check, wait, slot_acquired, slot_released

Database Migrations
-------------------
  - 0006_create_scatter_tables.exs: Barrier and result tables with indexes
    for efficient status queries and key lookups
  - 0007_create_signal_gate_tables.exs: Gate table with unique token hash
    index and partial index for pending timeout queries
  - 0008_update_oban_to_v11.exs: Oban schema update

Documentation
-------------
README.md has been extensively rewritten to clarify FlowStone's positioning
as an asset-first data orchestration library. New sections explain when to
use FlowStone versus alternatives like Handoff, Broadway, or plain Oban.
The comparison table helps users choose the right tool for their workload.

Examples added:
  - scatter_example.exs: Web scraping with parallel execution
  - signal_gate_example.exs: External task integration with callbacks
  - rate_limiter_example.exs: Rate limiting patterns and semaphores

Testing
-------
Comprehensive test suites added for all new modules:
  - scatter_test.exs: Key serialization, barrier lifecycle, gather semantics
  - signal_gate_test.exs: Token validation, gate state transitions, timeouts
  - rate_limiter_test.exs: Bucket limits, slot acquisition, error recovery

Breaking Changes
----------------
None. The Asset struct has new optional fields for scatter and signal gate
support, but existing code continues to work unchanged.
Assets 2
Loading