Measurement Harness

Measurement harness

How to get browser evidence about facebook/astryx without an hour of scaffolding. The operational copy is ~/astryx/review-loop-kit/measurement-harness.md, and the brief's STEP 5 sends you here the moment a review has to fill its VISUAL EVIDENCE slot — read it BEFORE cutting a worktree.

Skipping this costs ~35 minutes per task. That number is measured, not a guess: a three-line CSS fix took ~50 minutes of wall clock and ~36 of them were spent before the fix existed.

1. The BEFORE already exists — do not build one

~/astryx/worktrees/main is kept at origin/main, installed, built, and serving a static Storybook on port 6100.

~/astryx/gaps-loop/warm-main.sh     # idempotent; a no-op when already current

Run it first. Then measure the baseline against http://localhost:6100.

Never cut your own checkout of main. It is the same commit as the warm one and costs an install (~2.5 min) plus two builds. If you catch yourself creating a third worktree, stop and ask what it is for.

• new change → warm main is BEFORE, your worktree is AFTER. Two, not three.
• fix on an existing PR branch → "before" is that branch's head. Still two: measure before you edit, in the same worktree.

1b. Seed a worktree by CLONING node_modules, never a bare pnpm install

A cold install here costs ~79 s, and ~121 s each when three run at once. The cause is not pnpm and not the network: pnpm materialises ~61,000 files one syscall at a time, and this managed Mac's security stack inspects every one at 1.46 ms/file. (Proof it is not downloads: a cold install with an EMPTY store measured faster — 68 s — than one with a warm store.)

APFS clonefile() copies the whole tree in ONE syscall, copy-on-write, so it is near-instant and costs near-zero real disk until blocks diverge.

python3 ~/astryx/tools/fast-install.py ~/astryx/worktrees/<branch>
# optional: --donor <worktree-name>  (default: a worktree with an identical
#                                     lockfile, else main)

The script picks a lockfile-matched donor, clones, then runs a real pnpm install to reconcile the delta. Measured:

	plain	clone + reconcile
one install	79 s	11 s
three concurrent	121 s each	19 s each
donor on a different lockfile	79 s	15 s

By hand, if you need it: cp -c -R <donor>/node_modules <target>/node_modules, the same for each packages/*/node_modules, then pnpm install on top if the branch changed the lockfile.

The donor must be IDLE. Reading it is safe — copy-on-write means the target diverges without touching the source — but cloning a worktree mid-install gives you a half-written tree. main is the default donor precisely because nobody installs into it; do not point --donor at a worktree another agent is in.

Nothing global was changed to make this work: no .npmrc, no nodeLinker, no store surgery, and existing worktrees do not need reinstalling. Concurrent runs are safe — the three-concurrent row was measured, not assumed.

Do NOT symlink one shared node_modules into every worktree. Tested, fails three ways: pnpm install rejects the tree with ERR_PNPM_ABORTED_REMOVE_MODULES_DIR and would delete the shared copy out from under every other worktree; .modules.yaml records an absolute path, so one tree is only ever valid for one worktree; and lockfiles differ between branches (3 distinct hashes across 5 sampled), so a shared tree silently gives a branch the wrong deps.

2. In your worktree run storybook dev — never rebuild to re-measure

.storybook/main.ts aliases @astryxdesign/core (and lab, charts, richtext, the themes) straight to packages/*/src, and main.test.ts guards that invariant. So under storybook dev:

• no @astryxdesign/build build is needed for the components — Vite compiles source (but the Storybook config needs one; see the traps below)
• no storybook build between measurement rounds — edit, HMR reloads, re-run the probe

cd apps/storybook && npx --yes pnpm@11.10.0 exec storybook dev -p 6200

Two traps, both measured:

• pnpm -F @astryxdesign/storybook dev -p 6200 does not work — the package script already carries -p 6006, and your flag is appended after it, so the server either takes 6006 or dies. Run storybook dev directly, as above.
• A fresh worktree does need exactly one build, despite what §2 says: .storybook/main.ts imports @astryxdesign/build's dist, so the config cannot load without it. pnpm -F @astryxdesign/build build once. The no-build rule holds for the components — they alias to source — not for the Storybook config itself.

This is the biggest single saving, because measuring is never one-shot: a review finding changes the code and every number has to be retaken. Free with dev, another full cycle with a build.

Wait for HMR to settle, not for the port to answer. networkidle lies — the server responds while the previous render is still on screen, so a probe can read stale DOM and pass for the wrong build. Assert on something the change makes true (that is what ready is for below).

3. Do not write a Playwright probe from scratch

~/astryx/probe-kit/lib.cjs has the preamble every probe was re-implementing.

const {openStory, eachEngine, row} = require(
  process.env.HOME + '/astryx/probe-kit/lib.cjs',
);

const out = await eachEngine(['chromium'], async engine => {
  const {browser, page} = await openStory(engine, 'core-tablist--overflow', {
    port: 6200,
    ready: () => document.querySelector('nav[aria-label]') != null,
  });
  const v = await page.evaluate(() => /* measure */ 0);
  await browser.close();
  return v;
});
row('after', out);

openStory(engine, story, {port, viewport, ready, rtl, launch}). Two traps the kit now handles for you, both of which produced CONFIDENTLY WRONG numbers before they were found:

• RTL goes through Storybook's direction global, not document.dir. The preview decorator owns dir and passes it to the provider, so setting document.dir after load leaves the tree LTR while your probe reports RTL. openStory now sets the global and asserts it took.
• Headless never paints a scrollbar on macOS. Anything about scrollbars, gutters or layout shift needs {launch: {headless: false}} or it measures zero and looks like a pass.

Write your probe into ~/astryx/probe-kit/ when it is reusable, and say so in your report — 29 one-off probes had accumulated before this existed.

4. NEVER run the a11y audit locally

CI's pr-a11y runs on every PR touching components, reuses the Storybook artifact the build job made, scopes to the changed components, and fails on violations absent from .github/a11y-baseline.json.

• Only the AFTER matters. The baseline IS the before. Auditing the parent commit re-derives what the repo already knows.
• Push the draft first, review while CI runs. Do not serialise build → review → push.

By hand only when CI cannot answer it — e.g. a component the PR did not touch.

First, check CI actually RAN

Trusting CI instead of measuring locally only works if CI ran. A first-time fork contributor's workflows sit at action_required until a maintainer approves them, so a PR can look armed and healthy — auto-merge waiting politely — on checks that do not exist. A LOW CHECK COUNT is also the tell for a conflicting branch, so the count is the thing to read, never the colour.

gh api "repos/facebook/astryx/actions/runs?head_sha=<sha>" \
  --jq '.workflow_runs[] | "\(.name) \(.conclusion)"'
gh api -X POST repos/facebook/astryx/actions/runs/<id>/approve

Expect ~19 checks on a healthy PR against main. A PR based on a non-main branch (a stack) gets almost none by design — it picks them up when its base lands.

4b. Run the TARGETED tests, never the full suite

CI runs everything on every push, so a local full-suite run buys nothing and costs minutes — 7 of one run's 36 were exactly this, twice, because the first was not logged and had to be repeated.

• Run the test files your change touches, and the ones that cover the behavior.
• Push, then let CI be the full check. Fix what it turns up.
• If you do run something long, tee it to a file. A run whose output you have to reproduce to read is a run you did twice.

The exception is a suite you expect to be red for a reason CI cannot show you — say why before spending the minutes.

5. Scope evidence to the claim

Ask for the measurement that settles the question, plus a regression check on what the change could plausibly break. Not everything measurable.

• second engine: only when the fix depends on engine behaviour (overflow-clip-margin did; a padding does not)
• a matrix: only when the change interacts with variants, schemes or direction
• screenshots: required for anything visual, but before/after of the thing that changed — not a gallery

6. Remove your worktree when you are done

git worktree remove <path> (or rm -rf then git worktree prune). It touches only the bare repo's admin records — branches and commits are untouched.

101 stale worktrees had accumulated before anyone noticed, and the volume holding the pnpm store hit 96% full. Stale trees do not slow an install down (cost is per-worktree file count), but they eat the disk and they slow fast-install.py's donor scan, which walks the worktrees looking for a matching lockfile.

7. Two implementation agents on the Mac at once, maximum

Storybook builds compete for the same cores. If both agents are measuring, one.