[repro] main + WORKFLOW_H2_MULTIPLEX=0 - #3242
Closed
pranaygp wants to merge 2 commits into
Closed
Conversation
Isolation experiment for the hook-storm regression that arrives with main somewhere in e8934ad..b12f248. #3190 turned the events-path H2 agent from one in-flight request per connection into 100; this branch is current main (32ac8e7) with nothing changed but that feature's documented kill switch, so a storm here measures multiplexing and nothing else. The probe route exists because a flag that never reached the deployed function's runtime environment would look exactly like a flag that made no difference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Contributor
|
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (365 failed)astro (33 failed):
example (25 failed):
express (31 failed):
fastify (36 failed):
hono (34 failed):
nextjs-turbopack (48 failed):
nextjs-webpack (40 failed):
nitro (24 failed):
nuxt (28 failed):
sveltekit (36 failed):
vite (30 failed):
📋 Other (72 failed)e2e-vercel-prod-nest (35 failed):
e2e-vercel-prod-tanstack-start (37 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 📋 Other
✅ vercel-multi-region
|
Contributor
📊 Workflow Benchmarks❌ The benchmark run for Partial results from the failed run: commit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 440353ms → this run 415269ms (Δ -25084ms, -6%) 1020 steps (queue-hop) Cumulative STSO time: main 9664ms → this run 5302ms (Δ -4362ms, -45%) ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Contributor
There was a problem hiding this comment.
Pull request overview
Creates a disposable repro setup on top of current main to isolate whether disabling HTTP/2 multiplexing on the events path (WORKFLOW_H2_MULTIPLEX=0) changes the hook-storm corruption behavior, and adds a small probe route to verify the flag reaches the deployed runtime environment.
Changes:
- Set
WORKFLOW_H2_MULTIPLEX=0in theworkbench/nextjs-turbopackVercel deployment environment. - Add
GET /api/e2e-h2-flagto echo the runtime flag value and derived “multiplex enabled” boolean.
Reviewed changes
Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| workbench/nextjs-turbopack/vercel.json | Adds the kill-switch env var to disable H2 multiplexing for this workbench deployment. |
| workbench/nextjs-turbopack/app/api/e2e-h2-flag/route.ts | Adds a probe API route to confirm the env var is present at runtime before running the storm. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+12
to
+18
| export function GET() { | ||
| return Response.json({ | ||
| WORKFLOW_H2_MULTIPLEX: process.env.WORKFLOW_H2_MULTIPLEX ?? null, | ||
| // Mirrors `h2MultiplexEnabled()` in world-vercel's http-client. | ||
| multiplexEnabled: process.env.WORKFLOW_H2_MULTIPLEX !== '0', | ||
| }); | ||
| } |
`vercel.json`'s `env` block is documented as passing variables to the functions, but Vercel marks the property deprecated, and a flag that silently fails to arrive would make a null result read as "multiplexing is not the cause" — the one outcome where being wrong is expensive. instrumentation.ts now supplies the value if the deployment did not, and records which source won so the probe route can report it. That also tells us where mitigation advice should point if #3190 owns the regression. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Pranay Prakash <pranay.gp@gmail.com>
Contributor
Author
Result: the null branch of the registered predictionRun 30589591922,
The prediction registered above was ~33–48 of 200 if #3190 owned the regression, ~195 if it did not. 199. Stock Arrived at independently, the answer is #3198: @VaguelySerious's A/B pair from earlier today — One limitation worth stating: the flag's runtime value was not independently observed. The preview sits behind SAML, so This branch has served its purpose and should be closed rather than merged. |
Contributor
Author
|
Final reading, now that the rest of the evidence is in: this experiment landed on its null branch, and the null is the right answer. The prediction registered in the body before the run was: flag-off returns hook-storm to roughly 100–145 of 600 (≈33–48 of 200) if #3190 owns the regression, or stays at ≈195 of 200 if it does not. It stayed: 199 corrupted of 200 (30589591922), against 195/200 for unmodified The culprit is #3198 — matched-in-time A/B evidence, a forward-carried bisect, and a same-night control on the bisect base, all written up at #3198 (comment). The caveat on this null, stated plainly because it is what makes it trustworthy or not: the kill switch's arrival in the deployed runtime is structurally guaranteed — This PR should be closed rather than merged — it is a repro harness (a |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Not for merge. Disposable isolation experiment for the hook-storm regression.
What this branch is
Current
main(32ac8e73f) with exactly one behavioral change: the workbench app the storm runs against setsWORKFLOW_H2_MULTIPLEX=0in itsvercel.jsonenvblock. That is the documented kill switch for #3190 ([world-vercel] Make HTTP/2 actually multiplex on the events path).world-vercelreads the variable lazily at runtime insidecreateEventsDispatcher(), so a deployment-level env var reaches it — no build-time inlining is involved.Nothing is reverted and no package source is touched, so a storm here measures events-path multiplexing and nothing else.
Why
The hook-storm corruption rate steps from ~115–145 / 600 to saturation somewhere in
e8934ade9..b12f248b6, and it does not come from #3196 — two independent branches flipped on the samemainmerge (evidence in #3196). Fresh measurement on currentmain, same night, preview environment: hook-storm 195 corrupted / 0 completed of 200, hook-sleep control 50/50 clean (run 30588748635).#3190 is the prime suspect on mechanism. Before it, undici capped the events H2 agent at one in-flight request per connection — the
Clientconstructor coercespipelining, so H2'sdefaultPipelining: Infinitywas unreachable (nodejs/undici#4143). After it, up to 100 streams multiplex per connection. That is a step change in how concurrently one run's event reads and writes hit the wire, which is precisely the input distribution every hydration-latency race in the replay engine is sensitive to.Registered prediction (written before the run)
The comparison is against run 30588748635 — same night, same preview environment, same 200/50 shape. Not against the older 600-attempt baselines, which differ in both size and day.
The probe route
GET /api/e2e-h2-flagechoes the runtime value of the flag. A flag that silently failed to reach the deployed function would produce a "no change" result indistinguishable from "multiplexing is not the cause", so the flag is verified on the deployment before the storm is dispatched.How it is being run
Dispatched as a hook-first short run (
step_storm_attempts=0,hook_storm_attempts=200,attempts=50) rather than by label, so the answer lands in ~20 minutes instead of holding runners for a two-hour full storm. Label this PRevent-log-race-reproif a full storm is ever wanted here.Runs
5cc453ae7.2bb0858bb, before the instrumentation fallback existed, so a null result from it could not have been distinguished from the flag never arriving.