Measuring

Measuring instead of guessing

Every rule in reenact was measured against real data before it was written, and the scripts that did the measuring live in tools/. They are the reference implementation: when a rule needs to change, it is measured there first and ported to Go second, and the two are compared in the tests.

This is not ceremony. It repeatedly produced answers that contradicted the plan:

What the plan assumed	What measurement found
Truncating at "step N-1" works	API step numbers are not YAML indices — in 92% of jobs
A secrets keyring is needed	User secrets appear in 0 of 56 jobs' running steps
A cache server must be installed	act already ships one; verified working and persistent
A job name with " / " is a reusable call	15 of 30 such jobs were ordinary jobs
Reusable calls can never be faithful	10 of 14 are same-repo, so the file is the run's

Two of those cancelled planned work before it was built. One found a bug in shipped code.

The probes

script	measures
survey.py	how much of the real world is in scope
fixtures.py	downloads the offline corpus (58 runs, 1167 jobs, and the workflows they call)
stepname_probe.py	how GitHub renders job and step display names
matrix_probe.py	matrix expansion, include/exclude, leg naming
resolve_probe.py	end-to-end resolution against the corpus
fidelity_probe.py	how often each fidelity gap actually occurs
reusable_probe.py	reusable calls: ref kinds, inputs, drift
act_probe.sh	nine things about act's real behaviour
cache_probe.sh	act's built-in cache server, end to end
reusable_act_probe.sh	how act handles called workflows
dist_probe.sh	whether the release artifacts satisfy gh extension install's rule

After an act upgrade

Run the shell probes. They exist precisely to answer "which assumption did this break?":

tools/act_probe.sh
tools/cache_probe.sh
tools/reusable_act_probe.sh

Each prints PASS=n FAIL=n and exits non-zero if something moved.

The corpus

testdata/corpus/ holds 58 real failed runs with their jobs, steps and workflow files as they were at the run's commit. The test suite therefore runs offline and deterministically, and a change in GitHub's API surfaces as a failing test rather than a mystery in production.

go test ./...              # everything, no network
python3 tools/fidelity_probe.py   # the same numbers, computed independently