[benchmarks] Add SO (stream overhead) scenario + collapse PR-comment smallprint#3077
Conversation
…smallprint Adds a new SO benchmark metric modelling a haiku-size LLM token stream: a writer streams 300 fake 4-byte chunks paced at 100/s for 3s while a parallel reader drains the whole stream (same setup as SL). SO = end-to-end write+consume time beyond the 3s generation window, i.e. stream overhead/backpressure. The 3s window is single-sourced in the runner and passed to the workflow as args so the write span and subtraction can't drift. Also collapses the benchmark PR-comment definitions/methodology smallprint into a <details> dropdown, mirroring the "Previous results" section. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
🦋 Changeset detectedLatest commit: 9884f34 The changes in this PR will be included in the next version bump. This PR includes changesets to release 0 packagesWhen changesets are added to this PR, you'll see the packages that this PR includes changesets for and the associated semver types Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
🧪 E2E Test Results✅ All tests passed Summary
Details by Category✅ ▲ Vercel Production
✅ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
✅ 📋 Other
✅ vercel-multi-region
|
📊 Workflow Benchmarkscommit Backend:
ℹ️ Metric definitions & methodologyBest/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead: writer streams 300 4-byte chunks paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SL 50/60/125 · SO 250/500/1000 · STSO (1-20) 20/30/60 · STSO (101-120) 30/45/90 · STSO (1001-1020) 40/60/120 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
|
Could we make the text-delta workload deterministic but variable-length instead of repeatedly writing |
|
Could we also add a structured-delta variant resembling AI SDK events, for example |
|
No backport to This commit builds entirely on the new benchmark harness that exists only on To override, re-run the Backport to stable workflow manually via |
2 participants
What
Adds a new SO (Stream Overhead) benchmark scenario and metric, and hides the benchmark PR-comment smallprint behind a dropdown.
SO scenario
Models a haiku-size LLM streaming tokens: a writer emits 300 fake 4-byte chunks (~one token each) paced at 100 chunks/sec for 3 s while a parallel reader drains the whole stream. Setup, trigger, ready-barrier and deployment-side timing are otherwise identical to the existing SL scenario — the only difference is the reader stamps
doneAtafter the last chunk rather thanreadAton the first.SO = (doneAt - writtenAt) - 3000ms— the end-to-end write+consume time in excess of the modelled generation window, i.e. the stream's overhead/backpressure on top of the token rate. All timestamps are deployment-side (same methodology as the other metrics).The 3 s / 100-per-sec constants live once in the runner and are passed to
benchSoWorkflow(chunkCount, intervalMs)as args (likebenchSequentialStepsWorkflow), so the writer's paced span and the runner's subtraction can never drift. Targets are provisional, to be tightened oncemainbaselines land. No methodology-version bump (SO is a new metric; existing baselines stay comparable).PR-comment smallprint dropdown
The metric/scenario/target definitions and methodology paragraphs are now wrapped in a collapsed
<details><summary>ℹ️ Metric definitions & methodology</summary>, mirroring the existing "📜 Previous results" section, so the comment leads with the results table.Files
workbench/example/workflows/97_bench.ts—benchSoWorkflow+soReaderStep/soWriterStep+BenchStreamOverhead(symlinked intonextjs-turbopack/nitro-v3).packages/core/e2e/benchmark.test.ts— SO constants,runSoIteration, scenario +test(...), config/doc updates..github/scripts/render-benchmark-comment.mjs—soinMETRIC_LABELS/METRIC_ORDER; footer collapsed into<details>..github/scripts/render-benchmark-comment.test.js— collapsed-footer + SO-row coverage.Testing
node --test .github/scripts/render-benchmark-comment.test.js— 12/12 pass.@workflow/coretypecheck passes (covers the bench runner);97_bench.tscompiles via SWC;benchSoWorkflowresolves by name through the workbench registry.🤖 Generated with Claude Code