[world-vercel] Retry throttled (429) event writes in-process, honoring Retry-After - #3504
Conversation
🦋 Changeset detectedLatest commit: f5f6c37 The changes in this PR will be included in the next version bump. This PR includes changesets to release 17 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed E2E Test SummarySummary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 165564ms → this run 163269ms (Δ -2295ms, -1%) 📜 Previous results (1)943410dWed, 12 Aug 2026 20:32:51 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
There was a problem hiding this comment.
Pull request overview
Adds an in-process retry policy for event-write throttling (HTTP 429 / ThrottleError) in @workflow/world-vercel, honoring server-provided Retry-After and bounding total wait time to avoid pinning an invocation. This aligns event-write behavior with the intended semantics for throttle handling and reduces reliance on queue redelivery timing.
Changes:
- Implement budgeted,
Retry-After-honoring in-process retries forThrottleErroracross all event types inwithEventPostRetry. - Refactor transient retry eligibility into a helper and keep throttle waits independent of the transient retry attempt counter.
- Add unit tests covering throttle retry behavior (honor
retryAfter, default wait, eligibility-excluded events, and budget exhaustion), plus a changeset for a patch release.
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
| packages/world-vercel/src/event-retry.ts | Adds a separate 429 throttle retry path with a cumulative wait budget and Retry-After handling, without altering transient retry classification. |
| packages/world-vercel/src/event-retry.test.ts | Adds targeted unit tests validating throttle retry timing, applicability, and budget boundaries. |
| .changeset/throttle-in-process-retry.md | Declares a patch release note describing the new in-process 429 retry behavior and its motivation. |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
VaguelySerious
left a comment
VaguelySerious
left a comment
There was a problem hiding this comment.
Looks like the right approach to me
…g Retry-After (#3504) Signed-off-by: Nathan Rajlich <n@n8.io>
|
Backport PR opened against |
Context
The backend can reject an event write with a 429 and a
Retry-Afterof a few seconds (e.g. under write contention). The SDK mapped these toThrottleErrorbut never retried them in-process: the error escaped to the queue handler, and queue redelivery scheduling can delay the retry far beyond the requestedRetry-After— observed in practice as aretryAfter: 14throttle followed by a ~5-minute stall before the next delivery attempt.The intended behavior for
ThrottleErroris in-process retry.Change
withEventPostRetrynow retries 429s in-process:EVENT_RETRY_ELIGIBILITY-excluded ones (step_started,step_retrying,hook_received): a genuine application 429 is a definitive no-write — the server rejected the request outright — so the duplicate-row / attempt-double-count hazards that matrix protects against don't apply. This is the same reasoningSTREAM_RETRY_OPTIONSalready uses to retry 429 on stream PUTs.retryAfter(default 1s when absent).THROTTLE_RETRY_BUDGET_MS), addressing the "no attempt tracking → could spin until the flow route dies" concern. Once the budget can't cover the nextretryAfter, theThrottleErrorsurfaces and queue redelivery takes over exactly as before.errorForResponsemapsx-vercel-mitigated: challengeto a transportWorkflowWorldError, notThrottleError, so the original "let 429s pass through" rationale (never hot-loop against the firewall) is preserved.Each in-process throttle retry logs a visible
console.warn(it stalls the invocation for whole seconds, which would otherwise read as unexplained latency).Notes
Retry-Aftersizing may be worth revisiting now that the client waits it out in-process.Testing
event-retry.test.tscover: honoringretryAfter, applying to eligibility-excluded types, the no-retryAfterdefault, budget exhaustion (cumulative and single-wait-exceeds-budget), and budget independence from the transient allowance.pnpm vitest run src/event-retry.test.ts src/events-retry.test.ts src/events-v4-ws.test.ts— 53 passed.