[DO NOT MERGE] CI dry-run: QuickJS as the default workflow VM engine - #3253
Draft
TooTallNate wants to merge 2 commits into
Draft
[DO NOT MERGE] CI dry-run: QuickJS as the default workflow VM engine#3253TooTallNate wants to merge 2 commits into
TooTallNate wants to merge 2 commits into
Conversation
🦋 Changeset detectedLatest commit: 74bd053 The changes in this PR will be included in the next version bump. This PR includes changesets to release 16 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
Contributor
TooTallNate
added
the
event-log-race-repro
Contributor
🧪 E2E Test Results❌ Some tests failed ❌ Failed E2E Tests▲ Vercel Production (560 failed)astro-node (35 failed):
astro-quickjs (12 failed):
example-node (26 failed):
example-quickjs (20 failed):
express-node (26 failed):
express-quickjs (26 failed):
fastify-node (23 failed):
fastify-quickjs (21 failed):
hono-node (16 failed):
hono-quickjs (18 failed):
nextjs-turbopack-node (30 failed):
nextjs-turbopack-quickjs (17 failed):
nextjs-webpack-node (19 failed):
nextjs-webpack-quickjs (25 failed):
nitro-node (22 failed):
nitro-quickjs (19 failed):
nuxt-node (52 failed):
nuxt-quickjs (24 failed):
sveltekit-node (50 failed):
sveltekit-quickjs (19 failed):
vite-node (33 failed):
vite-quickjs (27 failed):
💻 Local Development (2 failed)astro-stable-node (1 failed):
sveltekit-stable-node (1 failed):
📋 Other (94 failed)e2e-vercel-prod-nest-node (33 failed):
e2e-vercel-prod-nest-quickjs (14 failed):
e2e-vercel-prod-tanstack-start-node (17 failed):
e2e-vercel-prod-tanstack-start-quickjs (30 failed):
E2E Test SummarySummary
Details by Category❌ ▲ Vercel Production
❌ 💻 Local Development
✅ 📦 Local Production
✅ 🐘 Local Postgres
✅ 🪟 Windows
❌ 📋 Other
✅ vercel-multi-region
|
VaguelySerious
left a comment
VaguelySerious
left a comment
Member
There was a problem hiding this comment.
AI review: blocking issues found
TooTallNate
force-pushed
the
quickjs-vm-default
branch
from
July 31, 2026 23:03
d156ea9 to
1ff3cc8
Compare
…ck into node:vm) - useQuickJSVm defaults to the QuickJS engine when neither the run's stamped executionContext.workflowVm nor WORKFLOW_VM specifies one; WORKFLOW_VM=node is the explicit node:vm opt-in. - CI matrix inverted to match: quickjs legs leave WORKFLOW_VM unset so the default-selection path is exercised end to end; node legs opt in explicitly (labels/artifacts unchanged via MATRIX_VM). - Entrypoint tests that assert node replay-loop internals against mock worlds pin WORKFLOW_VM=node (the quickjs path would instantiate a WASM VM per call). - Docs + changeset updated. Runs keep the engine stamped at start().
TooTallNate
force-pushed
the
quickjs-vm-default
branch
from
July 31, 2026 23:09
1ff3cc8 to
6af2c2c
Compare
Contributor
📊 Workflow Benchmarks❌ The benchmark run for Partial results from the failed run: commit Backend:
📈 STSO distribution vs main (inline / queue-hop histograms)1020 steps (inline) Cumulative STSO time: main 152131ms → this run 769388ms (Δ +617257ms, +406%) 1020 steps (queue-hop) Cumulative STSO time: main 2084ms → this run 4439ms (Δ +2355ms, +113%) 📜 Previous results (1)6af2c2cFri, 31 Jul 2026 23:40:21 GMT · run logs
ℹ️ Metric definitions & methodologyThe collapsed STSO distribution section above buckets every step gap of the sequential-steps run (not a sampled window), split by whether the step ending the gap ran inline — in the same warm process as the step before it, so the gap is pure framework overhead — or after a queue-hop — the first step of a fresh process, which pays queue dispatch, client reinit and event-log replay. Bars overlay the two runs: Best/P75/P90/P99 deltas compare against the most recent benchmark run on Metrics — TTFS: time to first step body (in-deployment start() → first step body, deployment clocks) · STSO: step-to-step overhead (gap between consecutive step bodies) · WO: workflow overhead (whole-run time outside step bodies, in-deployment anchored) · SL: stream latency (in-deployment write → read propagation, readAt - writtenAt) · SO: stream overhead (end-to-end write+consume time beyond the modelled generation window) Scenarios — step: one trivial no-op step, no stream; no hooks, so the run stays in turbo mode (in-process fast path) · stream: one streaming step; no hooks, so the run stays in turbo mode (in-process fast path) · hook + stream: registers a hook before one step, which exits turbo mode (dispatch path) · 1020 steps: 1020 trivial sequential steps; STSO is measured between consecutive steps in the given step ranges, and WO is the whole-run overhead outside step bodies · stream latency: parallel reader/writer steps on a dedicated stream; SL is the in-deployment write->read propagation (readAt - writtenAt) · stream overhead (text): writer streams 300 variable-length text token deltas paced at 100/s for 3s (a haiku-size LLM's token throughput) while a parallel reader drains the whole stream; SO is the end-to-end write+consume time beyond the 3s generation window (overhead/backpressure) · stream overhead (structured): same workload as stream overhead (text), but each delta is an AI-SDK-style structured object ({ type: 'text-delta', id, text }) instead of a raw string, so the SO gap vs the text scenario is the added serialization cost 🔴 marks a percentile over its target (within target is left unmarked). Targets (p75/p90/p99, ms) — TTFS 200/300/600 · SO 250/500/1000 All metrics are measured from deployment-side timestamps only. Runs are triggered by an in-deployment route that stamps the anchor ( Cold starts are kept in the numbers on purpose — they are part of real bursty-workload latency. The workbench deployment cold-starts the |
Contributor
|
Ran the event-log-race-repro storm locally against this branch on world-local and world-postgres (24 step-storm + 24 hook-storm per condition, concurrency 6,
¹ partial runs (21 and 19 landed of 48) — the launch budget cut them short. Caveat before reading the zero: under QuickJS not a single storm run completed — every run blew past the harness's 240s cap ( A fair comparison needs |
Contributor
|
Re-ran against the latest head (6af2c2c) with the harness run-timeout raised 240s → 600s so QuickJS runs get a fair chance to complete, plus a variant with the newest stack layer (#3263's
Performance (postgres, from event-log forensics):
Takeaways: (1) the corruption repro is alive on this branch under node; (2) a QuickJS corruption comparison is not yet meaningful — completion rate is 0/144 across every QuickJS condition even at 600s; (3) the ~8× interpreter gap is the first-order blocker, and the new serde layer (as merged here) introduces a wake/dormancy bug worth catching before the restack. Happy to share the harness setup ( |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Warning
Not intended to merge. This PR exists to exercise the full CI matrix — and the
event-log-race-reprostress workflow (via its label) — with QuickJS as the default engine, as a dress rehearsal for an eventual real default flip. It stays a draft; when the underlying stack (#3048 → #3049 → #3250 → #3251, plus #3263) is merged, a fresh flip PR will be cut against main with the learnings from this one.What it changes (for the dry run)
useQuickJSVmdefaults to the QuickJS engine when neither the run's stampedexecutionContext.workflowVmnorWORKFLOW_VMspecifies one;WORKFLOW_VM=nodeis the explicit node:vm opt-in.WORKFLOW_VMunset (exercising the default-selection path end to end); node legs opt in explicitly. Labels/artifacts unchanged viaMATRIX_VM.WORKFLOW_VM=node(see the comment at each pin).Findings so far (feeding the real flip PR)
event-limitred) was real and is fixed on QuickJS engine: inline step execution + WASM module caching #3049 (in-loop re-check + run_failed/MAX_EVENTS_EXCEEDED conversion).crypto, frozenprocess.env, loudIntl/locale guards); remaining differences are enumerated in the docs.start()— under a flipped default, unsetWORKFLOW_VMcurrently produces unstamped runs that would switch engines mid-run on rollback (review finding on this PR). The real flip PR must stamp the resolved engine.