[core] Derive correlation ids from per-kind sequences (opt-in) - #3301
Conversation
Correlation ids were the Nth draw of one monotonic ULID sequence per run, shared by steps, waits, hooks, attribute writes, abort controllers and stream ids. Every id was an ordinal over the whole run, so one extra draw of any kind renumbered every entity of every kind after it, and two replays that disagreed about a single sleep minted different ids for every step that followed. Each kind now draws from its own independent sequence, behind WORKFLOW_PER_KIND_CORRELATION_IDS=1. Also routes STABLE_ULID through the same generator. Serialization minted stream ids with no seed time, and monotonicFactory's increment branch returns encodeTime(lastTime), so one such call latched the host wall clock and every later id in the run carried a per-replay timestamp instead of fixedTimestamp. That fix applies in both modes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 7da5002 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
📊 Workflow Benchmarks❌ The benchmark run for Partial results from the failed run: commit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 148883ms → this run 137813ms (Δ -11070ms, -7%) 1020 steps (queue-hop) Cumulative STSO time: main 3591ms → this run 3567ms (Δ -24ms, -1%) 📜 Previous results (2)e49c053Mon, 03 Aug 2026 22:27:27 GMT · run logs
b82b39dMon, 03 Aug 2026 19:45:36 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (122 failed)astro (17 failed):
example (10 failed):
express (17 failed):
fastify (6 failed):
hono (16 failed):
nextjs-turbopack (9 failed):
nextjs-webpack (17 failed):
nitro (5 failed):
nuxt (5 failed):
sveltekit (8 failed):
vite (12 failed):
💻 Local Development (1 failed)astro-stable (1 failed):
📦 Local Production (1 failed)vite-stable (1 failed):
📋 Other (24 failed)e2e-vercel-prod-nest (10 failed):
e2e-vercel-prod-tanstack-start (14 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
❌ 💻 Local Development
❌ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 📋 Other
✅ vercel-multi-region
|
VaguelySerious
added
the
event-log-race-repro
|
(AI) Thanks for running both schemes and the baseline comparison — that changed the shape of this PR. On efficacy, you are right and I am not going to argue it. step-storm not moving is by construction (same-kind disagreement, explicitly out of scope), but hook-storm is where cross-kind absorption should have shown up and it did not move outside noise: 4/6 and 3/6 here against 4/6 on the flag-off baseline in #3273. I do not have production evidence that isolates the cross-kind class either. So the honest position is that this is a mechanism whose failure class is real in the code but not demonstrated by the harness built to measure it. That is the reason for the change in this push rather than a re-run: the default is off again. Shipping an upgrade-boundary scheme change on the strength of an argument, with a flat harness, was the wrong trade. Opt-in, it costs nothing to anyone who does not set the variable, and Everything else in this push is your review:
Core unit suite green in both schemes; |
|
Re-reviewed at e49c053 — this addresses everything substantive from my review (opt-in default, both changesets, the Three follow-ups on docs and test wiring, none blocking: 1.
This claim is fully true only for server-minted ids ( 2. The page currently says "This error indicates a bug in the Workflow SDK or Workflow server — not in your workflow code." Once this flag exists, there's an operational cause too: flipping 3. The local race-repro rig still runs the old scheme The commit message says the workbench flag makes "e2e and the race-repro harness exercise the new scheme," but that's true only for the Vercel-deployed lanes: |
| 'workflow': patch | ||
| --- | ||
|
|
||
| Add experimental `WORKFLOW_PER_KIND_CORRELATION_IDS=1`, which gives each kind of entity a workflow creates its own sequence of correlation IDs so an extra hook or sleep no longer renumbers every step after it. Off by default; a run must replay under the scheme that minted its IDs, so only turn it on while no runs are in flight unless your platform pins a run to the deployment it started on |
Signed-off-by: Peter Wielander <mittgfu@gmail.com>
…ation-ids # Conflicts: # packages/core/src/workflow.ts
|
No backport to The bulk of this commit is feature work: a new opt-in To override, re-run the Backport to stable workflow manually via |
Correlation ids are minted by the workflow VM and are the backend's identity gate: a conditional create on the id is what makes a duplicate write from a second live replay idempotent instead of additive. Today every id is the Nth draw of one monotonic ULID sequence per run, shared by steps, waits, hooks, attribute writes, abort controllers and stream ids alike. Every id is therefore an ordinal over the whole run, and one extra draw of any kind renumbers every entity of every kind after it.
That is the coupling this PR removes. Each kind draws from its own independent sequence, so two replays that agree about every step but disagree about one
sleep()still mint the same id for the Nth step. Under the shared sequence they mint different ids for every step after the sleep, so their writes append side by side instead of colliding, and the settled log ends up holding two names for one logical step. Only one of them can be consumed on the next replay; the other is fatal (onUnconsumedEvent→CORRUPTED_EVENT_LOG).Off by default, opt in with
WORKFLOW_PER_KIND_CORRELATION_IDS=1. Default-off is deliberate: a run must replay under the scheme that minted its ids, and on a platform that does not pin a run to the deployment that started it (world-postgres,world-local, any self-hosted process) a default flip would make the SDK upgrade itself the scheme change, failing every in-flight run's next replay. Turning the flag on has that same hazard, so the docs say to do it during a quiet window and to roll the value out fleet-wide at once rather than through a rolling deploy that leaves both values live. On Vercel skew protection removes the hazard, andworkbench/nextjs-turbopacksets the flag so e2e and the race-repro harness run the new scheme.What this is not
This is deliberately much less than #3179 (call-site-addressed ids). It does not make an id independent of ordinal position within its own kind: two replays that disagree about how many steps ran still mint different ids for the next step. It narrows the blast radius of a disagreement to the one kind that actually disagrees, and it preserves properties #3179 gives up:
hooks.listkeeps creation order among the hooks a workflow creates. [measurement] Call-site-addressed correlation ids #3179's hash-per-call-site reorders it, which broke thewebhookWorkflowe2e. The caveat:hookandabortHookare separate kinds and no world filters system hooks out of a listing, so a run that constructs an abort controller and creates its own hooks lists that one system hook at a position set by its kind's hash. The only consumer in this repo iswf inspect's display ordering.Ids remain syntactically valid ULIDs: 10 Crockford characters of
fixedTimestampplus a 16-character body derived per kind from a 128-bit hash of the run seed, then advanced withincrementBase32.Also here: a replay-determinism fix, unconditional
serialization.tsmints stream ids (and an abort holder's stream name andabrt_hook token) through theSTABLE_ULIDglobal and calls it with no seed time.ulid'smonotonicFactoryreturnsencodeTime(lastTime)on its increment branch, so a single such call latches the host wall clock intolastTime, and every id the run mints afterwards carries that timestamp instead offixedTimestamp. The timestamp differs on every replay, so any workflow that serialized a stream before creating further entities minted different correlation ids on each replay.STABLE_ULIDnow goes through the same generator, which always passesfixedTimestamp. This applies in both modes, and has its own changeset.Testing
packages/core/src/correlation-id.test.ts: determinism across replays, per-run distinctness, monotonicity within a kind, that extrahook/wait/stream/attr/abortdraws do not renumber steps, that an abort controller does not renumber user hooks, that every id decodes tofixedTimestampin both modes, and the flag's default.packages/core/src/correlation-id-replay.test.ts: the same properties seen through the primitives that actually mint ids, including consuming an event log authored with the derived id.packages/core/src/workflow.test.ts: arunWorkflow-level pin on theSTABLE_ULIDbinding site. Reverting that line to the raw monotonic factory fails it withReplayDivergenceError, which is the bug the fix removes.Docs Preview
WORKFLOW_PER_KIND_CORRELATION_IDS