Skip to content

Reduce Vercel E2E polling load - #3316

Merged
karthikscale3 merged 2 commits into
mainfrom
codex/reduce-e2e-polling-load
Aug 4, 2026
Merged

Reduce Vercel E2E polling load#3316
karthikscale3 merged 2 commits into
mainfrom
codex/reduce-e2e-polling-load

Conversation

@karthikscale3

@karthikscale3 karthikscale3 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

  • make the Run.returnValue polling interval configurable with WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS
  • set all Vercel E2E jobs to poll every 5 seconds
  • document and test the new runtime setting

Why

The Vercel E2E matrix was running many test variants concurrently, and each active Run.returnValue waited for completion by polling the Workflow API every second. That combined fan-out produced the recent api-workflow traffic spike attributed to Vercel Labs CI.

This keeps the SDK default at 1 second for existing users while reducing CI polling frequency. The tradeoff is up to roughly 4 seconds of additional completion-detection latency in these E2E jobs.

Validation

  • pnpm vitest run packages/core/src/runtime/runs.test.ts (22 tests)
  • pnpm --filter @workflow/core build
  • pnpm --filter @workflow/core typecheck
  • parsed .github/workflows/tests.yml as YAML
  • git diff --check

Note: validation ran on Node 25, while the repository declares support for Node 18/20/22/24; pnpm emitted an engine warning, but all checks above passed.

@changeset-bot

changeset-bot Bot commented Aug 4, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 2691538

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 16 packages
Name Type
@workflow/core Patch
@workflow/builders Patch
@workflow/cli Patch
@workflow/next Patch
@workflow/nitro Patch
@workflow/vitest Patch
@workflow/web-shared Patch
@workflow/web Patch
workflow Patch
@workflow/world-testing Patch
@workflow/astro Patch
@workflow/nest Patch
@workflow/rollup Patch
@workflow/sveltekit Patch
@workflow/vite Patch
@workflow/nuxt Patch

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@vercel

vercel Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
example-nextjs-workflow-turbopack Ready Ready Preview Aug 4, 2026 2:12am
example-nextjs-workflow-webpack Ready Ready Preview Aug 4, 2026 2:12am
example-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-astro-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-express-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-fastify-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-hono-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-nestjs-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-nitro-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-nuxt-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-sveltekit-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-tanstack-start-workflow Ready Ready Preview Aug 4, 2026 2:12am
workbench-vite-workflow Ready Ready Preview Aug 4, 2026 2:12am
workflow-docs Ready Ready Preview, v0 Aug 4, 2026 2:12am
workflow-swc-playground Ready Ready Preview Aug 4, 2026 2:12am
workflow-tarballs Ready Ready Preview Aug 4, 2026 2:12am
workflow-web Ready Ready Preview Aug 4, 2026 2:12am

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

📊 Workflow Benchmarks

commit 2691538 · Tue, 04 Aug 2026 02:25:17 GMT · run logs

Backend: vercel · app: nextjs-turbopack

Metric Scenario Best (ms) P75 (ms) P90 (ms) P99 (ms) Samples
TTFS step 242 (-77%) 💚 1355 🔴 (+21%) 🔻 1409 🔴 (+24%) 🔻 1780 🔴 (+22%) 🔻 30
TTFS stream 246 (+6.5%) 1369 🔴 (+26%) 🔻 1379 🔴 (+24%) 🔻 1603 🔴 (+26%) 🔻 30
TTFS hook + stream 396 (-19%) 💚 1545 🔴 (+11%) 1567 🔴 (+8.9%) 1647 🔴 (-7.7%) 30
STSO 1020 steps (inline) 98 (±0%) 136 (-11%) 159 (-13%) 333 (+13%) 1018
STSO 1020 steps (queue-hop) 2460 (-14%) 2460 (-14%) 2460 (-14%) 2460 (-14%) 1
WO 1020 steps 140127 (-5.8%) 140127 (-5.8%) 140127 (-5.8%) 140127 (-5.8%) 1
SL stream latency 84 (-24%) 💚 191 🔴 (-1.5%) 213 🔴 (-26%) 💚 606 🔴 (+57%) 🔻 30
SO stream overhead (text) 118 (-4.1%) 227 (+5.6%) 311 (+26%) 🔻 979 (+238%) 🔻 30
SO stream overhead (structured) 133 (+15%) 245 (+25%) 🔻 326 (+39%) 🔻 912 (+164%) 🔻 30
📈 STSO distribution vs main (inline / queue-hop histograms)

1020 steps (inline)

Cumulative STSO time: main 144885ms → this run 136335ms (Δ -8550ms, -6%)

   50-100 ms  ┃                         main   3  this   4    +1
  100-150 ms  ████████████████████░░░┃  main 722  this 883  +161
  150-200 ms  ██┃███                    main 234  this  97  -137
  200-250 ms  ┃                         main  37  this  17   -20
  250-300 ms  ┃                         main  12  this   3    -9
  300-350 ms  ┃                         main   5  this   5    +0
  350-400 ms  ┃                         main   4  this   2    -2
  400-450 ms  ┃                         main   0  this   3    +3
  500-550 ms  ┃                         main   0  this   1    +1
  600-650 ms  ┃                         main   1  this   0    -1
  650-700 ms  ┃                         main   0  this   1    +1
  800-850 ms  ┃                         main   0  this   1    +1
1100-1150 ms  ┃                         main   0  this   1    +1

1020 steps (queue-hop)

Cumulative STSO time: main 2849ms → this run 2460ms (Δ -389ms, -14%)

2000-2500 ms  ░░░░░░░░░░░░░░░░░░░░░░░┃  main 0  this 1  +1
2500-3000 ms  ┃███████████████████████  main 1  this 0  -1
ℹ️ Metric definitions & methodology

The collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: is main, marks where this run lands, bridges the gap when this run has more samples in a bucket.

Best/P75/P90/P99 deltas compare against the most recent benchmark run on main at the time of this run. 🔻 flags a delta worse than +15%, 💚 one better than −15%.

Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window)

Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost

🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000

All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor (clientStart) right before start(), so the CI runner’s request and its path through api.vercel.com sit outside every measured window. TTFS = in-deployment start() → first step body (turbo uses the in-process fast path, non-turbo the dispatch path), and includes the VQS dispatch hop plus any /flow cold start. STSO/WO are measured between step bodies on the deployment. SL is measured inside the workflow (parallel reader/writer steps), so it no longer includes the api.vercel.com read path.

Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the /flow invocation for a large fraction of runs, inflating P75+; the Best column shows the fastest (warm-start) sample for comparison.

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

🧪 E2E Test Results

Some tests failed

❌ Failed E2E Tests

💻 Local Development (11 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable
📦 Local Production (11 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable
🐘 Local Postgres (12 failed)

astro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

express-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

fastify-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

hono-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-turbopack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-canary-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nextjs-webpack-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nitro-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

nuxt-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

sveltekit-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

vite-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

📋 Other (7 failed)

e2e-local-dev-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-dev-tanstack-start-node (1 failed):

  • fetchWorkflow | wrun_41KZ58TW7X0GQ427GJNXD1K5YC

e2e-local-dev-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-postgres-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-postgres-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-prod-nest-stable-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

e2e-local-prod-tanstack-start-quickjs (1 failed):

  • hookMinRetentionWorkflow - terminal Hook cannot resume and its token stays unavailable

E2E Test Summary

Summary
Passed Failed Skipped Total
✅ ▲ Vercel Production 2780 0 496 3276
❌ 💻 Local Development 2986 11 435 3432
❌ 📦 Local Production 3123 11 454 3588
❌ 🐘 Local Postgres 3278 12 454 3744
✅ 🪟 Windows 156 0 0 156
❌ 📋 Other 2061 7 428 2496
✅ vercel-multi-region 27 0 0 27
Total 14411 41 2267 16719
Details by Category

✅ ▲ Vercel Production

App Passed Failed Skipped
✅ astro-node 127 0 29
✅ astro-quickjs 127 0 29
✅ example-node 127 0 29
✅ example-quickjs 127 0 29
✅ express-node 127 0 29
✅ express-quickjs 127 0 29
✅ fastify-node 127 0 29
✅ fastify-quickjs 127 0 29
✅ hono-node 127 0 29
✅ hono-quickjs 127 0 29
✅ nextjs-turbopack-node 152 0 4
✅ nextjs-webpack-node 152 0 4
✅ nextjs-webpack-quickjs 152 0 4
✅ nitro-node 127 0 29
✅ nitro-quickjs 127 0 29
✅ nuxt-node 127 0 29
✅ nuxt-quickjs 127 0 29
✅ sveltekit-node 146 0 10
✅ sveltekit-quickjs 146 0 10
✅ vite-node 127 0 29
✅ vite-quickjs 127 0 29

❌ 💻 Local Development

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
❌ express-stable-quickjs 129 1 26
✅ fastify-stable-node 130 0 26
❌ fastify-stable-quickjs 129 1 26
✅ hono-stable-node 130 0 26
❌ hono-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-stable-quickjs 155 1 0
✅ nitro-stable-node 130 0 26
❌ nitro-stable-quickjs 129 1 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

❌ 📦 Local Production

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
❌ express-stable-quickjs 129 1 26
✅ fastify-stable-node 130 0 26
❌ fastify-stable-quickjs 129 1 26
✅ hono-stable-node 130 0 26
❌ hono-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-canary-quickjs 136 1 19
✅ nextjs-webpack-stable-node 156 0 0
✅ nitro-stable-node 130 0 26
❌ nitro-stable-quickjs 129 1 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

❌ 🐘 Local Postgres

App Passed Failed Skipped
✅ astro-stable-node 130 0 26
❌ astro-stable-quickjs 129 1 26
✅ express-stable-node 130 0 26
❌ express-stable-quickjs 129 1 26
✅ fastify-stable-node 130 0 26
❌ fastify-stable-quickjs 129 1 26
✅ hono-stable-node 130 0 26
❌ hono-stable-quickjs 129 1 26
✅ nextjs-turbopack-canary-node 137 0 19
❌ nextjs-turbopack-canary-quickjs 136 1 19
✅ nextjs-turbopack-stable-node 156 0 0
❌ nextjs-turbopack-stable-quickjs 155 1 0
✅ nextjs-webpack-canary-node 137 0 19
❌ nextjs-webpack-canary-quickjs 136 1 19
✅ nextjs-webpack-stable-node 156 0 0
❌ nextjs-webpack-stable-quickjs 155 1 0
✅ nitro-stable-node 130 0 26
❌ nitro-stable-quickjs 129 1 26
✅ nuxt-stable-node 130 0 26
❌ nuxt-stable-quickjs 129 1 26
✅ sveltekit-stable-node 149 0 7
❌ sveltekit-stable-quickjs 148 1 7
✅ vite-stable-node 130 0 26
❌ vite-stable-quickjs 129 1 26

✅ 🪟 Windows

App Passed Failed Skipped
✅ nextjs-turbopack-node 156 0 0

❌ 📋 Other

App Passed Failed Skipped
✅ e2e-local-dev-nest-stable-node 130 0 26
❌ e2e-local-dev-nest-stable-quickjs 129 1 26
❌ e2e-local-dev-tanstack-start-node 129 1 26
❌ e2e-local-dev-tanstack-start-quickjs 129 1 26
✅ e2e-local-postgres-nest-stable-node 130 0 26
❌ e2e-local-postgres-nest-stable-quickjs 129 1 26
✅ e2e-local-postgres-tanstack-start-node 130 0 26
❌ e2e-local-postgres-tanstack-start-quickjs 129 1 26
✅ e2e-local-prod-nest-stable-node 130 0 26
❌ e2e-local-prod-nest-stable-quickjs 129 1 26
✅ e2e-local-prod-tanstack-start-node 130 0 26
❌ e2e-local-prod-tanstack-start-quickjs 129 1 26
✅ e2e-vercel-prod-nest-node 127 0 29
✅ e2e-vercel-prod-nest-quickjs 127 0 29
✅ e2e-vercel-prod-tanstack-start-node 127 0 29
✅ e2e-vercel-prod-tanstack-start-quickjs 127 0 29

✅ vercel-multi-region

App Passed Failed Skipped
✅ nextjs-turbopack 27 0 0

📋 View full workflow run

@karthikscale3
karthikscale3 marked this pull request as ready for review August 4, 2026 02:04
@karthikscale3
karthikscale3 requested review from a team, fantix and msullivan as code owners August 4, 2026 02:04
Comment thread .github/workflows/tests.yml Outdated
statuses: read
strategy:
fail-fast: false
max-parallel: 6

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure about this - it'll make CI take a total of 50 minutes to run, quite annoying

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah thought it might lower the load further. will skip for now

@karthikscale3
karthikscale3 merged commit e084e08 into main Aug 4, 2026
116 of 162 checks passed
@karthikscale3
karthikscale3 deleted the codex/reduce-e2e-polling-load branch August 4, 2026 02:30
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

No backport to stable for e084e08 (AI decision).

This commit introduces a new user-configurable runtime setting (WORKFLOW_RETURN_VALUE_POLL_INTERVAL_MS) that adds a config surface and a new internal export to @workflow/core, which is feature/enhancement work rather than a fix for a defect on stable. The motivation is reducing this repo's own CI polling traffic, not a correctness, crash, or resource-leak problem affecting stable users, whose existing 1-second default behavior is unchanged. The CI (tests.yml) tweak and docs addition only exist to exercise and document the new option, so they do not stand on their own as build/release plumbing fixes.

To override, re-run the Backport to stable workflow manually via workflow_dispatch and paste this commit SHA into the ref input:

e084e08ac0b4f1830659cb316df50d50962aada4

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants