Resilient step dispatch: parallelize step_created writes with queue publishes - #3365
Conversation
…_created + queue publish) Newly created steps are handed to the queue in parallel with their step_created event write, with the serialized input carried on the message (stepInput) so the queue consumer can idempotently re-ensure the event when the direct write failed transiently — mirroring resilient start (runInput) and resilient hook resume (hookInput). - @workflow/world: stepInput on WorkflowInvokePayload, CreateEventParams.viaStepDispatch, WorldCapabilities.resilientStepDispatch - core (node:vm): suspension handler publishes eligible steps alongside their create; the dispatch pass skips them (queuedStepCorrelationIds) - core (quickjs): dispatchPendingOps does the same for overflow steps; the ineligible fallback is now published in parallel too (removes the serial per-step enqueue loop) - consumer: on a redelivery, a stepInput-carrying message re-ensures step_created (marked viaStepDispatch) before executing - under an enforced precondition guard the parallel path requires backend cooperation (capabilities.resilientStepDispatch, declared by world-vercel): a 412-rejected step's in-flight dispatch is revoked server-side and its re-ensure refused - step dispatch/retry idempotency keys are step-identity-scoped (cid + hashed step name) so a revoked message for a reassigned correlation id cannot absorb the corrected schedule's dispatch - kill switch: WORKFLOW_RESILIENT_STEP_DISPATCH=0
🦋 Changeset detectedLatest commit: 23d70d9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 20 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
There was a problem hiding this comment.
Pull request overview
This PR implements resilient step dispatch for Workflow SDK: newly created steps can be queued with their serialized stepInput while the producer concurrently writes the step_created event, enabling the consumer to re-ensure step_created on redelivery when the direct write failed transiently. It also updates dispatch idempotency keys to be step-identity-scoped and adds capability gating + telemetry + docs.
Changes:
- Add
WorkflowInvokePayload.stepInputand supporting world contracts (viaStepDispatch,resilientStepDispatchcapability) to enable consumer-side re-ensure ofstep_created. - Parallelize step dispatch in both node VM suspension handling and QuickJS overflow dispatch, carrying serialized step input on the queue message when eligible.
- Update idempotency keys, telemetry conventions, docs, and add tests + changeset for the new behavior.
Reviewed changes
Copilot reviewed 17 out of 17 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| packages/world/src/queue.ts | Adds stepInput schema/types to queue payloads for resilient step dispatch. |
| packages/world/src/queue.test.ts | Adds a schema round-trip test for stepInput on step messages. |
| packages/world/src/interfaces.ts | Adds WorldCapabilities.resilientStepDispatch capability flag. |
| packages/world/src/events.ts | Adds CreateEventParams.viaStepDispatch marker for consumer re-ensure writes. |
| packages/world-vercel/src/index.ts | Declares backend capability support for resilient step dispatch under guard. |
| packages/world-vercel/src/events.ts | Threads viaStepDispatch through event creation metadata. |
| packages/world-vercel/src/events-v4.ts | Adds viaStepDispatch to v4 create-event metadata plumbing. |
| packages/core/src/telemetry/semantic-conventions.ts | Adds semantic convention attributes for resilient dispatch recovered/materialized. |
| packages/core/src/runtime/suspension-handler.ts | Implements parallel create+publish with stepInput payload and reports queued step CIDs. |
| packages/core/src/runtime/suspension-handler.test.ts | Adds tests covering eligibility gates and resilience semantics for step dispatch. |
| packages/core/src/runtime/quickjs-entrypoint.ts | Implements parallel create+publish for overflow steps and updates dispatch idempotency key usage. |
| packages/core/src/runtime/helpers.ts | Adds stepDispatchIdempotencyKey() helper and FNV-1a hashing for key scoping. |
| packages/core/src/runtime/constants.ts | Adds WORKFLOW_RESILIENT_STEP_DISPATCH kill-switch and payload size cap constant. |
| packages/core/src/runtime.ts | Consumer-side re-ensure of step_created from stepInput on redelivery; updates dispatch keys. |
| packages/core/src/runtime.test.ts | Adds tests for consumer re-ensure behavior on redelivery and legacy/no-stepInput messages. |
| docs/content/docs/v5/configuration/runtime-tuning.mdx | Documents WORKFLOW_RESILIENT_STEP_DISPATCH. |
| .changeset/resilient-step-dispatch.md | Adds changeset for @workflow/world, @workflow/world-vercel, and @workflow/core. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Review feedback: producers only attach stepInput when the dehydrated input is binary and the queue transport preserves bytes (CBOR), so a non-binary value means the payload was mangled in transit. Enforcing Uint8Array in StepDispatchInputSchema fails the message parse instead of silently writing non-binary data into a step_created, and types the consumer's re-ensure so the unchecked 'as SerializedData' cast goes away.
…he resilientStepDispatch capability lift) Review feedback (two P1s): backend-side revocation bookkeeping cannot carry the guard's correctness property across the queue side-channel — - nothing orders a slow guarded create's eventual 412 (the moment the backend learns the dispatch is poisoned and records the revocation marker) before the consumer's redelivery re-ensure, so attempt > 1 is a probabilistic mitigation, not a happens-before; and - a best-effort marker that fails open (Redis loss) cannot back a capability the SDK treats as a correctness attestation. Only sequencing the publish after the create gives the message a happens-after edge over the create's guard verdict, so the guard gate is now unconditional: worlds that enforce the precondition guard keep the sequential create-then-publish dispatch. The parallel resilient path remains for unguarded writes (the quickjs engine everywhere, and worlds without the guard). Removes WorldCapabilities.resilientStepDispatch and world-vercel's declaration; the viaStepDispatch flag is kept and re-documented as advisory (server-side defense-in-depth only). This also dissolves the reviewed dedupe hazard on the step-identity- scoped dispatch keys: with no 410-ack path in any real SDK flow, a message for a never-created step keeps redelivering until an entity exists, execution always hydrates input from the committed entity (never the message), and a name-mismatched stale start is skipped by the server's stepName fence.
… message-size cap 256 KB is the queue's inline-vs-S3 threshold, not a rejection limit (payloads above it spill to S3-backed storage transparently). The 128 KiB bound is a cost/latency choice — keep step messages on the inline path rather than paying an S3 double-hop for bytes that already live in the event log.
Conflict resolutions: - suspension-handler.ts: main inlined the EventCreator type (explicit createEvent/createGuarded signatures); kept this branch's added imports minus the removed type. - suspension-handler.test.ts / semantic-conventions.ts: additive on both sides — kept both. - world-vercel/events.ts: main restructured createEvent around a shared v4 input object with run_started / hook_received preload branches; re-applied this branch's viaStepDispatch spread onto the shared input so all three branches carry it.
karthikscale3
left a comment
karthikscale3
left a comment
There was a problem hiding this comment.
Reviewed alongside vercel/workflow-server#714. I do not see a correctness blocker in the current implementation; keeping guarded step dispatch sequential removes the serious stale-schedule race. Before merge, please update the PR description and rollout section because they still describe the removed resilientStepDispatch capability lift and imply the server PR must deploy first. I would also add or verify explicit coverage for a deterministic consumer re-ensure failure so an unusable queued message cannot burn all deliveries. The event-log preload optimization can remain a follow-up. Approving.
…ts its create Durabench parallel sweeps (guard-off, node engine) caught ~4-8% of fan-out runs stalling one branch for ~306s on the resilient dispatch path. Root cause: the consumer's step_created re-ensure was gated on metadata.attempt > 1, but world-vercel's failure-retry path re-enqueues a FRESH message whose attempt resets to 1 — so when a delivery beat the producer's parallel step_created write, every fast retry hit the same 'step not found' rejection with attempt 1, and the step only recovered when the ORIGINAL message's ~300s visibility-timeout redelivery finally arrived with attempt 2. The recovery is now in-band and attempt-independent: when a stepInput-carrying execution rejects with the step-missing signature (WorkflowWorldError, 404 or the local worlds' message shape), the consumer materializes the step_created from the message payload and retries the execution once within the same delivery. The eager attempt>1 ensure is kept as a round-trip saver on genuine redeliveries. Sweep effect expected: the 305-306s TTLS outliers disappear while the resilient path keeps its p50 win (1054ms vs 1425ms at 64 branches).
Baseline sweep results + a bug this PR's sweep caught (fixed in 74f7411)Ran the 4-version × {20, 64, 256}-branch parallel baseline sweep on durabench (node engine,
Root causeThe consumer's Fix (74f7411)Recovery is now in-band and attempt-independent: when a Re-running the guard-off cells to confirm the outliers are gone. (Separate finding, not this PR: all versions show a ~17s TTLS cliff at 256 branches — skew p99 ~9.5s. Tracking that as the next TTLS optimization target.) |
Conflict resolutions (main landed the SDK side of slot-mode event identity, specVersion 6 — #3389): - suspension-handler.ts: import union (main re-introduced EventCreator and added mergeReportedEvents; kept this branch's resilient-dispatch imports) and both result fields (queuedStepCorrelationIds + reportedEventCount). - suspension-handler.test.ts / runtime.ts: import unions (slotToEventId/maxEventSlot/settleEventSlotGap alongside this branch's stepDispatchIdempotencyKey).
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
|
No backport to This is feature work and a latency optimization, not a stability fix: it adds a new To override, re-run the Backport to stable workflow manually via |
Summary
Implements resilient step dispatch: when a suspension hands newly created steps to the queue, the runtime publishes each step's execution message in parallel with its
step_createdevent write instead of sequencing them, and the message carries the serialized step input (stepInput). If the direct write fails transiently (429 / 5xx / transport), the queue consumer idempotently re-ensures thestep_createdevent from the message payload before executing — the same durability pattern as resilient start (runInput) and resilient hook resume (hookInput).Motivation
Traces from a
parallelWorkflowfan-out (64 parallel steps) showed the dispatch phase dominating the invocation:step_createdwrites ran in parallel (~240ms) ✅Beyond latency, the create-then-publish sequencing meant a transient
step_createdwrite failure surfaced as a failed suspension pass and a full orchestrator redelivery.What changed
@workflow/worldWorkflowInvokePayload.stepInput— serialized step input on step-execution messagesCreateEventParams.viaStepDispatch— marks the consumer's re-ensureWorldCapabilities.resilientStepDispatch— backend cooperation attestation (see below)@workflow/core— producersPromise.allSettled([step_created write, queue publish w/ stepInput])per eligible step (all steps concurrent); queue failure is fatal (redelivery recovers, same as today), a transient create failure is swallowed and recovered by the consumer. Queued steps are reported back (queuedStepCorrelationIds) so the dispatch pass skips them.dispatchPendingOpsdoes the same for overflow steps, and the ineligible fallback path now publishes in parallel too — removing the serial per-step enqueue loop responsible for the 2.7s above.@workflow/core— consumerattempt > 1), astepInput-carrying message idempotently re-ensuresstep_created(markedviaStepDispatch) in parallel with the run fetch, then executes. First deliveries pay zero overhead: if the producer's write didn't land, the barestep_startedrejects with an error every world routes to redelivery, and attempt 2 materializes the step.Precondition-guard interaction
step_createdfrom a stale replay; a queue message carrying that step's payload must not let the consumer materialize what the guard rejected. The parallel path therefore requires the backend to attest cooperation viacapabilities.resilientStepDispatch(declared by@workflow/world-vercel): the backend revokes a 412-rejected step's in-flight dispatch (its re-ensure is refused and the message acked) and fences a barestep_startedon the entity's step name. Without the attestation, guard-enforcing worlds keep today's sequential dispatch. Worlds without the guard (world-local, world-postgres) don't need it.correlationId+ hashed step name) so a revoked in-flight message for a reassigned correlation id can never dedupe away the corrected schedule's legitimate dispatch. Safe across versions: queue messages are deployment-pinned, so one run never sees two key schemes.Kill switch:
WORKFLOW_RESILIENT_STEP_DISPATCH=0restores sequential dispatch (documented in runtime-tuning).Telemetry:
workflow.step.resilient_dispatch_recovered(producer) /workflow.step.resilient_dispatch_materialized(consumer) span attributes.Rollout
The Vercel backend's cooperating half (revocation markers +
viaStepDispatchhandling + step-name fence) must be deployed before this ships in an SDK release. Against an older backend, the flag is ignored and the guard-window protection is absent — hence the capability gate.Testing
viaStepDispatch, first-delivery zero-overhead, conflict-as-success, legacy messages)stepInputDocs Preview
WORKFLOW_RESILIENT_STEP_DISPATCH)/v5/docs/configuration/runtime-tuning#workflow_resilient_step_dispatch(link via the workflow-docs preview once the vercel[bot] comment appears)