[next] Fix dev HMR dropping edits that race a full rediscovery rebuild - #3529
Conversation
🦋 Changeset detectedLatest commit: 098a15a The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed
|
| Passed | Failed | Skipped | Total | |
|---|---|---|---|---|
| ✅ ▲ Vercel Production | 3474 | 0 | 738 | 4212 |
| ✅ 💻 Local Development | 3810 | 0 | 558 | 4368 |
| ✅ 📦 Local Production | 3810 | 0 | 558 | 4368 |
| ✅ 🐘 Local Postgres | 3810 | 0 | 558 | 4368 |
| ✅ 🪟 Windows | 312 | 0 | 0 | 312 |
| ✅ 🌐 Cross-language Conformance | 9 | 0 | 128 | 137 |
| ✅ vercel-multi-region | 27 | 0 | 0 | 27 |
| Total | 15252 | 0 | 2540 | 17792 |
Details by Category
✅ ▲ Vercel Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-node | 128 | 0 | 28 |
| ✅ astro-quickjs | 128 | 0 | 28 |
| ✅ example-node | 128 | 0 | 28 |
| ✅ example-quickjs | 128 | 0 | 28 |
| ✅ express-node | 128 | 0 | 28 |
| ✅ express-quickjs | 128 | 0 | 28 |
| ✅ fastify-node | 128 | 0 | 28 |
| ✅ fastify-quickjs | 128 | 0 | 28 |
| ✅ hono-node | 128 | 0 | 28 |
| ✅ hono-quickjs | 128 | 0 | 28 |
| ✅ nest-node | 128 | 0 | 28 |
| ✅ nest-quickjs | 128 | 0 | 28 |
| ✅ nextjs-turbopack-node | 153 | 0 | 3 |
| ✅ nextjs-turbopack-quickjs | 153 | 0 | 3 |
| ✅ nextjs-webpack-node | 153 | 0 | 3 |
| ✅ nextjs-webpack-quickjs | 153 | 0 | 3 |
| ✅ nitro-node | 128 | 0 | 28 |
| ✅ nitro-quickjs | 128 | 0 | 28 |
| ✅ nuxt-node | 128 | 0 | 28 |
| ✅ nuxt-quickjs | 128 | 0 | 28 |
| ✅ python-node | 8 | 0 | 148 |
| ✅ sveltekit-node | 147 | 0 | 9 |
| ✅ sveltekit-quickjs | 147 | 0 | 9 |
| ✅ tanstack-start-node | 128 | 0 | 28 |
| ✅ tanstack-start-quickjs | 128 | 0 | 28 |
| ✅ vite-node | 128 | 0 | 28 |
| ✅ vite-quickjs | 128 | 0 | 28 |
✅ 💻 Local Development
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 📦 Local Production
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 🐘 Local Postgres
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ astro-stable-node | 130 | 0 | 26 |
| ✅ astro-stable-quickjs | 130 | 0 | 26 |
| ✅ express-stable-node | 130 | 0 | 26 |
| ✅ express-stable-quickjs | 130 | 0 | 26 |
| ✅ fastify-stable-node | 130 | 0 | 26 |
| ✅ fastify-stable-quickjs | 130 | 0 | 26 |
| ✅ hono-stable-node | 130 | 0 | 26 |
| ✅ hono-stable-quickjs | 130 | 0 | 26 |
| ✅ nest-stable-node | 130 | 0 | 26 |
| ✅ nest-stable-quickjs | 130 | 0 | 26 |
| ✅ nextjs-turbopack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-turbopack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-turbopack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-stable-quickjs | 156 | 0 | 0 |
| ✅ nextjs-webpack-canary-node | 137 | 0 | 19 |
| ✅ nextjs-webpack-canary-quickjs | 137 | 0 | 19 |
| ✅ nextjs-webpack-stable-node | 156 | 0 | 0 |
| ✅ nextjs-webpack-stable-quickjs | 156 | 0 | 0 |
| ✅ nitro-stable-node | 130 | 0 | 26 |
| ✅ nitro-stable-quickjs | 130 | 0 | 26 |
| ✅ nuxt-stable-node | 130 | 0 | 26 |
| ✅ nuxt-stable-quickjs | 130 | 0 | 26 |
| ✅ sveltekit-stable-node | 149 | 0 | 7 |
| ✅ sveltekit-stable-quickjs | 149 | 0 | 7 |
| ✅ tanstack-start-node | 130 | 0 | 26 |
| ✅ tanstack-start-quickjs | 130 | 0 | 26 |
| ✅ vite-stable-node | 130 | 0 | 26 |
| ✅ vite-stable-quickjs | 130 | 0 | 26 |
✅ 🪟 Windows
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack-node | 156 | 0 | 0 |
| ✅ nextjs-turbopack-quickjs | 156 | 0 | 0 |
✅ 🌐 Cross-language Conformance
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ python | 9 | 0 | 128 |
✅ vercel-multi-region
| App | Passed | Failed | Skipped |
|---|---|---|---|
| ✅ nextjs-turbopack | 27 | 0 | 0 |
📊 Workflow Benchmarkscommit Backend:
Streams
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 440013ms → this run 150170ms (Δ -289843ms, -66%) 📈 CRTT drill-down vs main (RTT distributions & profiles)RTT over stream progress (avg per tenth of stream, bars scaled min→max): RTT by chunk size (avg per log size bin, ~160B → ~12KB serialized, bars scaled min→max): Delivery jitter over stream progress (avg positive CDV per tenth of stream, bars scaled min→max): 📜 Previous results (4)f715b96Fri, 14 Aug 2026 21:10:45 GMT · run logs
bddc74fFri, 14 Aug 2026 19:17:31 GMT · run logs
bbfb581Thu, 13 Aug 2026 23:43:22 GMT · run logs
b59f23aThu, 13 Aug 2026 22:47:36 GMT · run logs
ℹ️ Metric definitions & methodologyStreams: writer/reader sustained rates (steady window, 10% trimmed each side), first-chunk RTT (the stream-open path, before any buffering/backpressure), CRTT percentiles, and worst delivery stall (CDV max). Cells are medians across iterations; per-run values in the artifacts. No 🔴/🟢 marks until targets attach. The collapsed STSO distribution section above buckets every step gap, split inline (same warm process — pure framework overhead) vs queue-hop (fresh process — dispatch, reinit, replay). The collapsed CRTT drill-down: per-variant RTT histograms (fixed log bins, Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body) · Fan-out TTFS: fan-out time to first step (in-deployment start() → first of the parallel step bodies to complete) · Fan-out TTLS: fan-out time to last step (in-deployment start() → last of the parallel step bodies to complete, i.e. when the Promise.all resolves) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · CRTT: chunk round-trip time (per-chunk write → read latency, one clock domain: deployment → stream backend → same deployment) · CDV: chunk delay variation / delivery jitter (inter-arrival gap minus inter-write gap per seq-adjacent pair; skew-free; the row is each run's MAX positive value, so one stall moves it) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · Promise.all(100 steps): 100 trivial no-op steps started together in a single Promise.all; Fan-out TTFS is the first of them to complete and Fan-out TTLS the last, both from the in-deployment clientStart, so their gap is the spread the runtime adds across the fan-out · paced control (100/s, 60B): the control: 300 tiny (~60B) deltas metronome-paced at 100/s — zero workload structure, so it reads the transport floor and flush cadence, and disambiguates transport-wide vs workload-specific when a replay row moves · size sweep (100/s, 160B-12KB): same pacing as the control with deltas padded in rotation across seven log-spaced sizes (~160B–12KB) — rotation decouples size from stream position, so it isolates whether chunk size causes latency · replay gateway-gpt-5.4-nano-2000t (1x): raw provider SSE cadence captured at the AI gateway boundary (gpt-5.4-nano, the most popular gateway model; per-token deltas p50 208B = the modal production chunk size), replayed exactly as measured — the typical customer's workload; its CDV is the typical customer's real delivery jitter · replay eve-gpt-5.6-sol-2000t (1x): a captured eve turn (gpt-5.6-sol, the most-used demanding eve model; ~2000 output tokens = production p50 turn length) replayed exactly as measured — eve's envelope protocol re-ships the cumulative message so sizes ramp 142B→13KB; the demanding outlier tenant's reality · replay eve-gpt-5.6-sol-2000t (2x): the same eve capture at 2x — the headroom/stress row; real fast-tier models emit the same chunk sizes at proportionally higher rate, so time compression is a faithful speed model · first chunk (pooled): every run's seq-0 RTT pooled across all stream scenarios — the first chunk precedes any workload differentiation, so pooling samples one shared stream-open path with exact percentiles Replay cadences (semantic sha256) — eve-gpt-5.6-sol-2000t 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 All timestamps are deployment-side; runs are triggered in-deployment, so the CI runner and api.vercel.com sit outside every measured window. TTFS = Cold starts stay in the numbers (real bursty-workload latency, inflates P75+); Best is the warm floor. |
Sim WorldSimulated world deterministic testing for races. Traces 🟠 Mint-ordered log — 3 fail of 41 total
Full trace: 🟢 Append-only log — 0 fail of 41 total
Full trace: |
b59f23a to
bbfb581
Compare
bddc74f to
f715b96
Compare
A full rebuild reads sources twice: once when the bundler consumes them and once when refreshSourceSnapshots() re-anchors the classifier baseline from disk afterwards. An edit landing between those reads (the rebuild takes multiple seconds in real apps) was absorbed into the baseline without ever being built, so its queued watcher event classified as a no-op and the change never reached the manifest. This is the dominant CI flake: the nextjs-webpack local-dev e2e HMR fuzz test writes a new step definition right after triggering a full rediscovery, and when the write lands mid-rebuild the added step never appears in the manifest, timing out after 300s (~60 attempt-1 job failures over the last 10 days, failing every day). pinBaselinesAcrossFullRebuild() owns the fix structurally: it re-reads the relevant files from disk immediately before the rebuild callback runs and restores those values for files present in both maps after the refresh, so a mid-rebuild edit still diffs against what the rebuild consumed while duplicate watcher events for content the rebuild already consumed diff equal instead of cascading. The capture deliberately reads disk rather than cloning the live baseline map: writes landing in the capture window (test-teardown restores, multi-flush bursts) are content the imminent build consumes anyway, and a zero-read clone formulation un-absorbs them into chains of follow-up full rebuilds once real rebuilds take multiple seconds. That formulation was tried and reverted after CI reproduced the chains; the tradeoff is one serial read of the relevant set (~150-250ms at ~250 files) per full rediscovery. Files first discovered by the in-flight rebuild keep their post-build baseline: a new file the build missed has no baseline at all and its add event forces the follow-up rebuild. A create-then-edit within one rebuild window can still be absorbed - this narrows that pre-existing race rather than closing it. The initial-build path has the same two-read shape with no watcher yet attached; knowingly out of scope and documented at the call site. Signed-off-by: Alex Langenfeld <alex.langenfeld@vercel.com>
|
No backport to This is a genuine correctness fix (dropped HMR edits / CI flake), but it depends entirely on machinery that does not exist on To override, re-run the Backport to stable workflow manually via |
2 participants
Summary & Motivation
A full rebuild reads sources twice — once when the bundler consumes them, once when the classifier baseline is refreshed from disk afterwards — so an edit landing in between was absorbed into the baseline without being built, and its queued watcher event classified as a no-op.
pinBaselinesAcrossFullRebuildpins baselines across the rebuild instead: pre-build values for files present in both maps, and the snapshotsclassifyRebuildread for the files that triggered the rebuild, so routine duplicate watcher events still diff equal rather than cascading into another full rebuild.This is the largest ongoing CI flake — the nextjs-webpack local-dev e2e HMR fuzz test writes a step definition right after triggering a rediscovery and times out at 300s when the write lands mid-rebuild (~60 attempt-1 job failures in 10 days). For files the in-flight rebuild discovered for the first time this narrows a create-then-edit race rather than closing it; the initial-build startup window has the same shape and is out of scope, documented at the call site.
Test Plan
Unit tests cover the absorb race and the duplicate-event no-op; the previously-flaky HMR fuzz e2e passed 3/3 locally against nextjs-webpack where main reproduced the manifest drop, and a mid-rediscovery edit rig went from 2/6 dropped on base to 0/6.