Two tests out of the inner loop, and the suite reports its own wall time

[kmoneil]

Stacked on #5, #4, #3. Base is se-032-measure-lane; GitHub retargets as those merge. Independent of the other three in substance -- it touches no source file.

Why

Two tests were 17.9 s of a 58.8 s suite, which is 31%:

Test	Time
test_limits.py::TestTheScriptRuns::test_the_table	9.01 s
test_limits.py::TestTheScriptRuns::test_the_json	8.91 s
next slowest	2.45 s

Both run scripts/limits.py --quick as a subprocess. The script does real timing work, which is the point of it, and --quick already skips the 100,000-item runs. They differ only in output format: --json emits JSON instead of the table, so neither invocation can serve both.

The suite was inside its budget with no headroom: CLAUDE.md asks for local feedback under a minute and 58.00 s is under a minute by two seconds. A slow suite changes engineering behaviour long before it fails a gate -- people stop running it, batches get larger, and feedback time becomes the real constraint.

Before and after, same machine

	Tests	Wall time
before	2,281 passed, 6 skipped	58.00 s
after, default selection	2,414 passed, 6 skipped, 2 deselected	39.6 s
after, --runslow	2,416 passed, 6 skipped	57.6 s

The "after" rows carry 135 more tests, because #3, #4 and #5 landed first. Like for like on one commit: the default selection is 39.6 s and the full one 57.6 s, so the two tests are still 18 s and still 31% -- they are just no longer 31% of what a developer waits for.

Marked and deselected from the inner loop; run in full wherever green has to mean something

scripts/lanes.py passes --runslow in fast, compat and sdist.

sdist was not on the original list and should have been: it runs the shipped suite from an unpacked distribution, so without the flag a lane that exists to prove the distribution is self-testing would have quietly stopped running two of its tests. Verified by running the lane: 2,350 passed, 18 skipped.

test_regression_limits_the_script_still_runs_in_ci asserts both ends -- the lane's command carries the flag and CI invokes the lane. Either alone would pass while the tests stopped running. A marker that quietly stopped running in CI would be strictly worse than the ten seconds it saved.

pytest --runslow runs the lot; pytest -m slow runs only these two.

Deselected rather than skipped

A skip is a line of output per test claiming something was considered and not done. A deselection is one number in the summary saying the selection was smaller, which is what actually happened. It also leaves an explicit -m slow working without --runslow, because a marker expression is a request rather than a default.

Two cheaper-looking fixes, rejected and recorded

Adding a --both mode to scripts/limits.py so one invocation serves both tests halves the cost, and means changing a published script's command line so that a test can run faster. That is the wrong way round: the script is documented in docs/performance.md as the thing a reader runs to reproduce the table on their own machine, and its interface belongs to them.

Reducing what --quick measures is ruled out. Cutting sample counts is how a measurement becomes a flake, and this repository has already fixed one flake in exactly that file caused by taking a median of three instead of a minimum of five. test_regression_limits_the_samples_were_not_quietly_reduced now reads the script and asserts both the count and the min, so it cannot come back through a performance change.

Coverage is unchanged, and that is measured

99.65% line on the default selection, identical to the number with both tests selected. They run the script as a subprocess, which coverage does not trace, so they contributed no src/ coverage to begin with.

The suite reports its own wall time now

Printed on every full run, with a 240 s tripwire beside it -- roughly four times the number it protects. A ceiling near the measurement fails on a busy machine, and a check that fails for a reason nobody can act on gets deleted rather than investigated. What this catches is somebody adding a test that takes a minute.

It arms only for the default selection: -k, -m and a path argument all disarm it, because a subset is faster by construction and timing one against a whole-suite ceiling would be measuring nothing.

It has been watched failing. test_the_runtime_tripwire_can_actually_fail calls the hook with the clock set ten thousand seconds in the past and asserts the exit status changes.

Regression battery

• test_regression_limits_the_script_still_runs_in_ci
• test_regression_limits_the_marked_tests_are_the_two_that_were_measured -- beyond the plan. A marker that spreads takes coverage out of the inner loop a test at a time and nothing else would notice.
• test_regression_limits_the_samples_were_not_quietly_reduced
• test_the_runtime_tripwire_is_loose_enough_not_to_be_noise, ..._can_actually_fail, ..._stays_quiet_for_a_subset, ..._reports_the_time_even_when_it_passes

What this is not

A licence to speed up the suite generally. The other 2,412 tests run in about 40 s, which is 17 ms each. The three next-slowest are load-bearing and they stay: a denial-of-service regression test that has to do real work to prove anything, and two differential tests that generate expressions across the whole allowlist.

Checks

• ruff check, ruff format --check, mypy --strict clean
• Default selection: 2,414 passed, 6 skipped, 2 deselected, 39.6 s
• Full selection: 2,416 passed, 6 skipped, 57.6 s
• Coverage: 99.65% line, unchanged
• sdist lane run locally and passing from the distribution with the flag
• No source file touched