Skill: benchmark a toolchain change across build, CI, and runtime

[MajorLift]

Size: L | Domain: platform (or a new build)

---

Problem

A toolchain change moves three independent numbers, and measuring one misleads about the others:

Surface	React Compiler	thread-loader	oxlint
Build time (dev loop)	slower — extra transform	faster — parallel pool	faster than ESLint
CI time	slower	depends on runner cores	faster
Runtime performance	faster — memoized output	neutral — output unchanged	neutral

Judge React Compiler on build time and you reject it. Judge it on runtime and you hide a dev-loop tax every engineer pays on every rebuild. The webpack enablement is the flagship case — it moved all three at once, and there was no consistent way to say by how much.

This skill publishes into two evidence lanes — D7 Build & rebuild duration A/B for the dev loop and G6 CI job-duration delta for CI — which encode the confounds below as their falsifiers. Both are in review on #84, not yet on the default branch. C5 benchmark A/B covers runtime only.

---

Why this is hard to measure honestly

Each falsifier below silently produces a favourable result, which is what makes ad-hoc measurement worse than none:

• Cache state differs between arms. Warm versus cold is the largest confound and the easiest to introduce by running the new arm second.
• Worker-pool warmup. thread-loader pays pool startup on the first build and not on rebuilds, so first-build numbers understate it and steady-state numbers overstate it.
• Core count. Parallel loaders scale with available cores, so a laptop result does not transfer to a CI runner, or the reverse.
• Watch rebuild versus cold build are different questions; conflating them answers neither.
• CI build reuse. get-requirements.yml skips jobs when build output matches base, so a measured CI speedup can be a reuse-rate change rather than a faster build.
• Release adoption curve. On the runtime side, a window sampled before rollout plateaus mixes versions and reads as improvement.

The thread-loader integration also carries a coupling worth testing rather than trusting: loader options must stay JSON-serializable for the pool, and the pool must be ordered before the SWC rules.

---

Solution

A paired A/B protocol per surface, each with its confound named and controlled:

1. Dev loop — cold and warm reported separately, cache cleared explicitly between arms, N runs reported with spread rather than a mean alone.
2. CI — same workflow, same runner class, arms at a fixed commit differing only by the flag, with reuse rate stated so a skip is not read as a speedup.
3. Runtime — p75/p95 over matched windows after adoption plateaus, with the version filter stated.

The verdict is a table across all three. A change that costs build time and buys runtime is a trade; the skill prices it rather than approving it.

---

Acceptance criteria

• A protocol per surface, each stating its controls
• A falsifier per arm: cache state, pool warmup, core count, CI build reuse, adoption window
• Worked example on the React Compiler / thread-loader enablement, real numbers, both arms
• Reports spread, not central tendency alone; states the smallest detectable effect when a result is null
• Renders no ship verdict — the trade-off belongs to whoever owns the dev loop

---

Notes

May want splitting once the shape is clear: three protocols under one skill is a lot, and the dev-loop and CI halves may separate cleanly from the runtime half.

Related but distinct from a CI-health cluster. This prices a known change across surfaces; diagnosing an unexplained CI regression is the other direction and wants different instruments.

---

Tracked in Jira as AIENG-95.