You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This commit was created on GitHub.com and signed with GitHub’s verified signature.
Release
Overview
This release completes the implementation of Step-Level Concurrency Control for PartitionStep, formalizes Partition Execution Semantics, expands automated Failure Matrix integration tests, and introduces OpenTelemetry observability across worker executions.
What's Changed
1. Step-Level Concurrency Control (PartitionStep)
concurrency Parameter Support: Added partition.concurrency configuration in JSL step definitions, allowing developers to set maximum concurrent partition worker bounds per step.
Safe Defaults: Defaults to concurrency: 1 (enforcing "Serial-First" execution safety) when unspecified.
Semaphore Throttling: Integrated golang.org/x/sync/semaphore inside PartitionStep.Execute to throttle active workers and prevent resource exhaustion (e.g., DB connection starvation, memory spikes).
OpenTelemetry Instrumentation: Added worker-level OpenTelemetry span tracking for worker start, success, and error events with safe NoOp fallbacks when unconfigured.
⚠️ Developer Guidance & Migration Notes
JSL Configuration: When defining type: partition steps in JSL, configure concurrency based on target workload requirements (e.g., concurrency: 8). Unconfigured steps default to 1.
Observability: Partition worker traces will now automatically export through configured OpenTelemetry collectors.