Skip to content

feat(web): the Evidence Matrix, read rather than computed (#708 view 2) - #747

Merged
eaitbrahim merged 1 commit into
mainfrom
feat-708-evidence-matrix
Sep 7, 2026
Merged

feat(web): the Evidence Matrix, read rather than computed (#708 view 2)#747
eaitbrahim merged 1 commit into
mainfrom
feat-708-evidence-matrix

Conversation

@eaitbrahim

Copy link
Copy Markdown
Contributor

Closes #708. Closes the honest cockpit milestone.

The last of #708's four views, and the one that waited for an engine record instead of being built on a guess.

Why it is a read

Building the matrix on request costs 11.9 / 12.9 / 14.3 seconds per session on the real ledger — ~39 s of CPU for three sessions, on a page the console re-polls every 15 s. And over the ledger as a whole it doesn't run at all: build_matrix requires synchronous columns, so a PBO is only ever defined within a session whose trials share a bar count, and a page can't pick that scope without inventing an operator's decision.

So #726 made trials pbo record every field of its PBOResult, and this reads them. The console displays results an operator ran and never runs one on their behalf — pinned by a test that makes build_matrix raise if it's called.

An unrun matrix is not an empty one

Three states, which is why the report carries candidate_sessions at all: no ledger · a ledger with columns and no recorded run · a ledger with runs. The middle one is the state this ships in, and the only one that can be acted on.

The guidance names a session that actually has columns. keel trials pbo --session all is the obvious thing to print and would filter to a session literally called "all", find nothing, and refuse — teaching an operator that the page doesn't know what it's talking about. So the report counts the trials build_matrix would accept and names one of those. On the tracked ledger that's pbo-grid-entry-lookback-2026-07-20, one of three genuine candidates.

It deliberately doesn't check synchronicity, which would mean reading every series — most of the cost this module exists to avoid. Suggesting a session that might turn out ragged, and getting a clear refusal from the command, is a much smaller harm than a page that costs twelve seconds to render.

PBO carries no judgement

It's the one figure a reader will want graded, and grading it is what the rail refuses: a high PBO beside a flat, positive OOS scatter is the good outcome — a broad plateau of near-identical configurations produces high PBO by construction. trials pbo's own closing sentence says to read it alongside the degradation slope, never alone, and both cross plainly so a reader can.

The dominance flags do carry a state, because they're already verdicts — and they're three-valued: False says the distribution did not dominate, which is not what an absent field says.

⛔ No sortable column on the route and no sort key in the view. A matrix ordered by PBO is a leaderboard of overfitting scores, and cscv.py forbids PBO as a ranking key in its own source.

Two tests that weren't testing anything

test_a_recorded_run_stops_telling_the_operator_to_run_one seeded a ledger with no columns, so suggested_session was empty and the invocation was empty whatever the rule said — it passed against a page that prints the command forever. And the bool-is-an-int guard was pinned through a fixture no writer can produce; it's checked at the reader now.

Both found by mutation. So was a stale __pycache__: the mutant MIN_COLUMNS_FOR_A_RUN = 21 is the same byte length, so restoring the source within one mtime tick left Python reusing the compiled mutant. Worth knowing for any same-length mutation.

Gates

🤖 Generated with Claude Code

https://claude.ai/code/session_01KZZxmspQXe5qJ9FAsG13s6

The last of #708's four views, and the one that waited for an engine record
instead of being built on a guess.

WHY IT IS A READ

Building the matrix on request costs 11.9 / 12.9 / 14.3 seconds per session on
the real ledger -- ~39 s of CPU for three sessions, on a page the console
re-polls every 15 seconds. And over the ledger as a whole it does not run at all:
`build_matrix` requires synchronous columns, so a PBO is only ever defined WITHIN
a session whose trials share a bar count, and a page cannot pick that scope
without inventing an operator's decision.

So #726 made `trials pbo` record every field of its `PBOResult`, and this reads
them. The console displays results an operator ran and never runs one on their
behalf -- pinned by a test that makes `build_matrix` raise if it is called.

AN UNRUN MATRIX IS NOT AN EMPTY ONE

Three states, and the middle one is why the report carries `candidate_sessions`
at all: no ledger, a ledger with columns and no recorded run, and a ledger with
runs. The middle one is the state this ships in, and it is the only one that can
be acted on.

THE GUIDANCE NAMES A SESSION THAT ACTUALLY HAS COLUMNS

`keel trials pbo --session all` is the obvious thing to print and would filter to
a session literally called "all", find nothing, and print a refusal -- teaching
an operator that the page does not know what it is talking about. So the report
counts the trials `build_matrix` would ACCEPT (a per-bar series, not
`series_missing`) and names one of those. On the tracked ledger that is
`pbo-grid-entry-lookback-2026-07-20`, one of three genuine candidates.

It deliberately does not check SYNCHRONICITY, which would mean reading every
series -- most of the cost this module exists to avoid. Suggesting a session that
might turn out ragged, and getting a clear refusal from the command, is a much
smaller harm than a page that costs twelve seconds to render.

PBO CARRIES NO JUDGEMENT

It is the one figure a reader will want graded and grading it is exactly what the
rail refuses: a high PBO beside a flat, positive OOS scatter is the GOOD outcome
-- a broad plateau of near-identical configurations produces high PBO by
construction. `trials pbo`'s own closing sentence says to read it alongside the
degradation slope, never alone, and both cross plainly so a reader can.

The dominance flags DO carry a state, because they are already verdicts, and they
are three-valued: `False` says the distribution did not dominate, which is not
what an absent field says.

⛔ No sortable column on the route and no sort key in the view. A matrix ordered
by PBO is a leaderboard of overfitting scores, and `cscv.py` forbids PBO as a
ranking key in its own source.

TWO TESTS THAT WERE NOT TESTING ANYTHING

`test_a_recorded_run_stops_telling_the_operator_to_run_one` seeded a ledger with
no columns, so `suggested_session` was empty and the invocation was empty
whatever the rule said -- it passed against a page that prints the command
forever. And the bool-is-an-int guard was pinned through a fixture no writer can
produce, so it asserted nothing; it is checked at the reader now, which is the
honest way to pin a defensive check.

Both found by mutation. So was a stale `__pycache__`: the mutant
`MIN_COLUMNS_FOR_A_RUN = 2` -> `1` is the same byte length as the original, so
restoring the source within one mtime tick left Python reusing the compiled
mutant. Worth knowing for any same-length mutation.

Eleven mutants killed: the matrix built on request, pre-#726 gauntlet rows read
as matrices, a dominance flag read as a column count, an absent flag read as a
denial, a backfilled session suggested, a one-column session suggested, the
command still printed after a run, PBO gaining a judgement, the section dropped
from the view, the table gaining a sort control, and the route becoming sortable.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KZZxmspQXe5qJ9FAsG13s6
@eaitbrahim
eaitbrahim merged commit 17c4579 into main Sep 7, 2026
4 checks passed
@eaitbrahim
eaitbrahim deleted the feat-708-evidence-matrix branch September 7, 2026 10:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[web] Research Hub /research: trials ledger, evidence matrix, gauntlet, slippage universe

1 participant