[world-local] [test-only] Extend test budget on Windows runner for local storage test - #3503
Conversation
…ner can meet
`Unit Tests (windows-latest)` has been red on main since 2026-08-11 with
Error: Test timed out in 120000ms.
❯ src/storage.test.ts:1284:7
That is `returns the complete preload when run_started is retried`. It writes
a thousand events sequentially, one file write each, then asserts the retry
returns the whole log.
It is a marginal budget rather than a step regression. Same 245 tests either
side of the boundary, whole-file duration on the Windows runner went 143491ms
(last green) to 154816ms (first red), 8% apart. An added fs op per write would
show a far larger jump. On macOS the writes run at ~1ms each and the test
finishes in about a second, so it only ever bites on that runner.
Raise the budget to 300s and say why in the test, including that batching the
writes with `Promise.all` measures slower rather than faster: the writers then
contend for the same event slot and re-probe.
The count is load-bearing and was a bare 999/1001. It sits one past the event
cache so the preload cannot be served from cached entries alone and has to
read at least one back from disk. Below the ceiling the test still passes and
silently stops covering that fallback, so export the ceiling and size the test
from it.
Unit Tests (windows-latest) is an input to E2E Required Check, so this has been
failing the required aggregate on every open PR.
🦋 Changeset detectedLatest commit: 104c2f9 The changes in this PR will be included in the next version bump. This PR includes changesets to release 18 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (6 failed)nextjs-turbopack-node (1 failed):
nextjs-webpack-node (1 failed):
nextjs-webpack-quickjs (1 failed):
nuxt-quickjs (1 failed):
sveltekit-node (1 failed):
sveltekit-quickjs (1 failed):
🐘 Local Postgres (1 failed)sveltekit-stable-quickjs (1 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
❌ 🐘 Local Postgres
✅ 🪟 Windows
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 204038ms → this run 185007ms (Δ -19031ms, -9%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 6 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
VaguelySerious
changed the title
VaguelySerious
changed the title
|
No backport to This is a CI-stability fix, but the code it touches does not exist on To override, re-run the Backport to stable workflow manually via |
VaguelySerious
changed the title
VaguelySerious
changed the title
…ner can meet (#3503) Co-Authored-By: shalabhchaturvedi-7802 <shalabh.chaturvedi@vercel.com>
1 participant
Unit Tests (windows-latest)has been red onmain. One of its two independentfailures is
packages/world-local/src/storage.test.ts, reported at lines 1284and 1353 across main runs 31604115603, 31533009514 and 31616799260.
returns the complete preload when run_started is retriedwrites 999 eventssequentially, then asserts the preload. On the Windows runner each write costs
orders of magnitude more than locally, and the whole file drifted past its
budget: a green run took 143491ms and a failing run 154816ms with the same 245
tests, 8% apart.
events.createis linear locally at ~0.95ms across n = 250,500, 1000, 2000, so nothing got slower per event. It is a marginal budget, not
a step regression.
Two things that look like fixes and are not:
the writers contend for the same event slot and re-probe.
The 1000 is load-bearing: it is the event cache's entry ceiling, so the test
is the only thing covering the read-past-cache fallback. Below the ceiling it
goes on passing while silently covering nothing.
So: export
MAX_CACHED_EVENT_ENTRIESand size the test from it (ceiling + 1)rather than restating the number, and raise the timeout. The comment records
why the count is what it is and why batching is not the cheaper option.
Windows CI on this branch confirms world-local now passes entirely,
Test Files 15 passed (15).The other failure family,
packages/core/src/events-consumer.test.tsat lines1005 and 1123, is not addressed here. It has its own fix in a follow-up PR.