Skip to content

[world-vercel] [builders] Add ENFORCE_STRICT_CONCURRENCY option to limit flow route concurrency to one#2193

Draft
VaguelySerious wants to merge 9 commits into
stablefrom
peter/enforce-strict-concurrency
Draft

[world-vercel] [builders] Add ENFORCE_STRICT_CONCURRENCY option to limit flow route concurrency to one#2193
VaguelySerious wants to merge 9 commits into
stablefrom
peter/enforce-strict-concurrency

Conversation

@VaguelySerious
Copy link
Copy Markdown
Member

@VaguelySerious VaguelySerious commented Jun 1, 2026

Adds an opt-in WORKFLOW_ENFORCE_STRICT_CONCURRENCY=1 environment variable, which limits flow route concurrency to one at the queue level. Step routes are unaffected. The behavior is off by default.

This is done in two halves:

  • Build time (all builders): when enabled, the flow route's experimentalTriggers config gets maxConcurrency: 1. Centralized via a new getWorkflowQueueTrigger() getter in @workflow/builders, consumed by the Build Output API builder.
  • Runtime (@workflow/world-vercel only): when enabled, flow messages are sent to a per-run physical VQS topic (__wkf_workflow_<name>_<runId>) while the logical queueName in the message wrapper stays unchanged, so the handler and sleep/delay re-enqueue path resolve the same per-run topic.

Potential edge-cases

  • Observability cardinality: per-run topics create one distinct queueName per run, so the Vercel Queues observability UI/metrics (which group by queueName) get unbounded cardinality for projects that enable this. Gated behind the opt-in flag, but worth flagging to Queues o11y owners before broad rollout. Per-run drill-down still exists via the x-vercel-workflow-run-id header.
  • Both halves must agree: the variable is read at both build time and runtime and must be set as a single project-level env var that applies to both. Build-on / runtime-off would put maxConcurrency: 1 on the shared __wkf_workflow_<name> topic and serialize all runs of a workflow through one slot. On Vercel a project env var covers build + runtime, which avoids this; documented in the Vercel World docs.
  • Topic length: __wkf_workflow_<workflowName>_<runId> with a long namespaced/scoped workflow name plus the run id could approach VQS topic-name length limits. Should confirm the cap and add a hash-suffix fallback if needed before recommending broadly.

Testing

The feature is off by default. Anyone running a reproduction against strict concurrency must explicitly set WORKFLOW_ENFORCE_STRICT_CONCURRENCY=1 as a project-level environment variable before building and deploying, so both the generated flow trigger and the runtime queue routing opt in to the behavior.

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Jun 1, 2026

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview, Comment Jun 1, 2026 10:52am
example-nextjs-workflow-webpack Ready Ready Preview, Comment Jun 1, 2026 10:52am
example-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-astro-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-express-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-fastify-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-hono-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-nitro-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-nuxt-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-sveltekit-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-tanstack-start-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workbench-vite-workflow Ready Ready Preview, Comment Jun 1, 2026 10:52am
workflow-docs Ready Ready Preview, Comment, Open in v0 Jun 1, 2026 10:52am
workflow-swc-playground Ready Ready Preview, Comment Jun 1, 2026 10:52am
workflow-tarballs Ready Ready Preview, Comment Jun 1, 2026 10:52am
workflow-web Ready Ready Preview, Comment Jun 1, 2026 10:52am

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented Jun 1, 2026

🧪 E2E Test Results

Some tests failed

Summary

Passed Failed Skipped Total
✅ 💻 Local Development 970 0 86 1056
✅ 📦 Local Production 970 0 86 1056
✅ 🐘 Local Postgres 970 0 86 1056
✅ 🪟 Windows 88 0 0 88
❌ 🌍 Community Worlds 136 92 0 228
✅ 📋 Other 492 0 36 528
Total 3626 92 294 4012

❌ Failed Tests

🌍 Community Worlds (92 failed)

mongodb (14 failed):

  • hookWorkflow is not resumable via public webhook endpoint
  • webhookWorkflow
  • sleepingWorkflow
  • outputStreamWorkflow no startIndex (reads all chunks)
  • outputStreamWorkflow negative startIndex (reads from end)
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns correct index after stream completes
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns -1 before any chunks are written
  • outputStreamWorkflow - getTailIndex and getStreamChunks getStreamChunks returns same content as reading the stream
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions
  • writableForwardedFromWorkflowWorkflow
  • writableForwardedFromStepWorkflow
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously
  • pages router sleepingWorkflow via pages router
  • resilient start: addTenWorkflow completes when run_created returns 500

redis (9 failed):

  • hookWorkflow is not resumable via public webhook endpoint
  • sleepingWorkflow
  • outputStreamWorkflow negative startIndex (reads from end)
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns correct index after stream completes
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns -1 before any chunks are written
  • outputStreamWorkflow - getTailIndex and getStreamChunks getStreamChunks returns same content as reading the stream
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously
  • pages router sleepingWorkflow via pages router
  • resilient start: addTenWorkflow completes when run_created returns 500

turso-dev (1 failed):

  • dev e2e should rebuild on imported step dependency change

turso (68 failed):

  • addTenWorkflow
  • addTenWorkflow
  • wellKnownAgentWorkflow (.well-known/agent)
  • should work with react rendering in step
  • promiseAllWorkflow
  • promiseRaceWorkflow
  • promiseAnyWorkflow
  • importedStepOnlyWorkflow
  • readableStreamWorkflow
  • hookWorkflow
  • hookWorkflow is not resumable via public webhook endpoint
  • webhookWorkflow
  • sleepingWorkflow
  • parallelSleepWorkflow
  • nullByteWorkflow
  • workflowAndStepMetadataWorkflow
  • outputStreamWorkflow no startIndex (reads all chunks)
  • outputStreamWorkflow positive startIndex (skips first chunk)
  • outputStreamWorkflow negative startIndex (reads from end)
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns correct index after stream completes
  • outputStreamWorkflow - getTailIndex and getStreamChunks getTailIndex returns -1 before any chunks are written
  • outputStreamWorkflow - getTailIndex and getStreamChunks getStreamChunks returns same content as reading the stream
  • outputStreamInsideStepWorkflow - getWritable() called inside step functions
  • writableForwardedFromWorkflowWorkflow
  • writableForwardedFromStepWorkflow
  • fetchWorkflow
  • promiseRaceStressTestWorkflow
  • error handling error propagation workflow errors nested function calls preserve message and stack trace
  • error handling error propagation workflow errors cross-file imports preserve message and stack trace
  • error handling error propagation step errors basic step error preserves message and stack trace
  • error handling error propagation step errors cross-file step error preserves message and function names in stack
  • error handling retry behavior regular Error retries until success
  • error handling retry behavior FatalError fails immediately without retries
  • error handling retry behavior RetryableError respects custom retryAfter delay
  • error handling retry behavior maxRetries=0 disables retries
  • error handling catchability FatalError can be caught and detected with FatalError.is()
  • error handling not registered WorkflowNotRegisteredError fails the run when workflow does not exist
  • error handling not registered StepNotRegisteredError fails the step but workflow can catch it
  • error handling not registered StepNotRegisteredError fails the run when not caught in workflow
  • hookCleanupTestWorkflow - hook token reuse after workflow completion
  • concurrent hook token conflict - two workflows cannot use the same hook token simultaneously
  • hookDisposeTestWorkflow - hook token reuse after explicit disposal while workflow still running
  • stepFunctionPassingWorkflow - step function references can be passed as arguments (without closure vars)
  • stepFunctionWithClosureWorkflow - step function with closure variables passed as argument
  • closureVariableWorkflow - nested step functions with closure variables
  • spawnWorkflowFromStepWorkflow - spawning a child workflow using start() inside a step
  • health check (queue-based) - workflow and step endpoints respond to health check messages
  • health check (CLI) - workflow health command reports healthy endpoints
  • pathsAliasWorkflow - TypeScript path aliases resolve correctly
  • Calculator.calculate - static workflow method using static step methods from another class
  • AllInOneService.processNumber - static workflow method using sibling static step methods
  • ChainableService.processWithThis - static step methods using this to reference the class
  • thisSerializationWorkflow - step function invoked with .call() and .apply()
  • customSerializationWorkflow - custom class serialization with WORKFLOW_SERIALIZE/WORKFLOW_DESERIALIZE
  • instanceMethodStepWorkflow - instance methods with "use step" directive
  • crossContextSerdeWorkflow - classes defined in step code are deserializable in workflow context
  • stepFunctionAsStartArgWorkflow - step function reference passed as start() argument
  • cancelRun - cancelling a running workflow
  • cancelRun via CLI - cancelling a running workflow
  • pages router addTenWorkflow via pages router
  • pages router promiseAllWorkflow via pages router
  • pages router sleepingWorkflow via pages router
  • hookWithSleepWorkflow - hook payloads delivered correctly with concurrent sleep
  • sleepInLoopWorkflow - sleep inside loop with steps actually delays each iteration
  • sleepWithSequentialStepsWorkflow - sequential steps work with concurrent sleep (control)
  • importMetaUrlWorkflow - import.meta.url is available in step bundles
  • metadataFromHelperWorkflow - getWorkflowMetadata/getStepMetadata work from module-level helper (#1577)
  • resilient start: addTenWorkflow completes when run_created returns 500

Details by Category

✅ 💻 Local Development
App Passed Failed Skipped
✅ astro-stable 82 0 6
✅ express-stable 82 0 6
✅ fastify-stable 82 0 6
✅ hono-stable 82 0 6
✅ nextjs-turbopack-canary 69 0 19
✅ nextjs-turbopack-stable 88 0 0
✅ nextjs-webpack-canary 69 0 19
✅ nextjs-webpack-stable 88 0 0
✅ nitro-stable 82 0 6
✅ nuxt-stable 82 0 6
✅ sveltekit-stable 82 0 6
✅ vite-stable 82 0 6
✅ 📦 Local Production
App Passed Failed Skipped
✅ astro-stable 82 0 6
✅ express-stable 82 0 6
✅ fastify-stable 82 0 6
✅ hono-stable 82 0 6
✅ nextjs-turbopack-canary 69 0 19
✅ nextjs-turbopack-stable 88 0 0
✅ nextjs-webpack-canary 69 0 19
✅ nextjs-webpack-stable 88 0 0
✅ nitro-stable 82 0 6
✅ nuxt-stable 82 0 6
✅ sveltekit-stable 82 0 6
✅ vite-stable 82 0 6
✅ 🐘 Local Postgres
App Passed Failed Skipped
✅ astro-stable 82 0 6
✅ express-stable 82 0 6
✅ fastify-stable 82 0 6
✅ hono-stable 82 0 6
✅ nextjs-turbopack-canary 69 0 19
✅ nextjs-turbopack-stable 88 0 0
✅ nextjs-webpack-canary 69 0 19
✅ nextjs-webpack-stable 88 0 0
✅ nitro-stable 82 0 6
✅ nuxt-stable 82 0 6
✅ sveltekit-stable 82 0 6
✅ vite-stable 82 0 6
✅ 🪟 Windows
App Passed Failed Skipped
✅ nextjs-turbopack 88 0 0
❌ 🌍 Community Worlds
App Passed Failed Skipped
✅ mongodb-dev 5 0 0
❌ mongodb 57 14 0
✅ redis-dev 5 0 0
❌ redis 62 9 0
❌ turso-dev 4 1 0
❌ turso 3 68 0
✅ 📋 Other
App Passed Failed Skipped
✅ e2e-local-dev-nest-stable 82 0 6
✅ e2e-local-dev-tanstack-start-stable 82 0 6
✅ e2e-local-postgres-nest-stable 82 0 6
✅ e2e-local-postgres-tanstack-start-stable 82 0 6
✅ e2e-local-prod-nest-stable 82 0 6
✅ e2e-local-prod-tanstack-start-stable 82 0 6

📋 View full workflow run


Some E2E test jobs failed:

  • Vercel Prod: failure
  • Local Dev: success
  • Local Prod: success
  • Local Postgres: success
  • Windows: success

Check the workflow run for details.

⚠️ Community world tests failed (non-blocking):

  • Community Worlds: failure

Check the workflow run for details.

Comment thread .claude/worktrees/consistent-reads-stress Outdated
Signed-off-by: Peter Wielander <peter.wielander@vercel.com>
Forces ENFORCE_STRICT_CONCURRENCY behaviour on regardless of the env var so
all CI e2e jobs exercise per-run flow topics + maxConcurrency:1. Skips the
three off-by-default unit assertions. Drop this commit before merge.
@VaguelySerious VaguelySerious force-pushed the peter/enforce-strict-concurrency branch from b4bceac to 68a058d Compare June 1, 2026 09:30
@changeset-bot
Copy link
Copy Markdown

changeset-bot Bot commented Jun 1, 2026

🦋 Changeset detected

Latest commit: 4baa79d

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 17 packages
Name Type
@workflow/world-vercel Minor
@workflow/builders Minor
@workflow/next Minor
@workflow/sveltekit Patch
@workflow/cli Patch
@workflow/core Patch
@workflow/web Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/nitro Patch
@workflow/rollup Patch
@workflow/vite Patch
@workflow/vitest Patch
workflow Patch
@workflow/world-testing Patch
@workflow/web-shared Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

…issing field

A failed WorkflowRun exposes its reason as `error: { code, message }` and has no
top-level `errorCode`, so the poller's `classifyFailure(runData.errorCode)` was
always passing `undefined` — collapsing every polled failure to an
uncategorised, detail-less `other`. Read `runData.error.code`/`.message` so
USER_ERROR/RUNTIME_ERROR/CORRUPTED_EVENT_LOG are classified correctly and the
regression row shows why the run failed.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit d2a59d4)
…t-concurrency

# Conflicts:
#	packages/core/e2e/event-log-race-repro.test.ts
"@workflow/sveltekit": patch
---

Add opt-in `ENFORCE_STRICT_CONCURRENCY` env var. When set to `1`, flow (orchestrator) routes are limited to one invocation per run at a time via a per-run queue topic and `maxConcurrency: 1` on the flow trigger. Step routes are unaffected.
Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
Add opt-in `ENFORCE_STRICT_CONCURRENCY` env var. When set to `1`, flow (orchestrator) routes are limited to one invocation per run at a time via a per-run queue topic and `maxConcurrency: 1` on the flow trigger. Step routes are unaffected.
Add opt-in `ENFORCE_STRICT_CONCURRENCY` env var, which limits flow route concurrency to one in Vercel environments, at a small queue latency cost.

@vercel
Copy link
Copy Markdown
Contributor

vercel Bot commented Jun 1, 2026

Deployment failed with the following error:

Creating the Deployment Timed Out.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

event-log-race-repro Run the event log race reproduction job

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants