Skip to content

testsuite: measure T1/T2 so a verdict can be trusted — gates, loadout, abandon bound, nav stamp, stall kinds - #13

Merged
dsvensson merged 14 commits into
qw-ctf:mainfrom
Xerialen:main
Aug 1, 2026
Merged

testsuite: measure T1/T2 so a verdict can be trusted — gates, loadout, abandon bound, nav stamp, stall kinds#13
dsvensson merged 14 commits into
qw-ctf:mainfrom
Xerialen:main

Conversation

@Xerialen

Copy link
Copy Markdown
Contributor

Claude (on behalf of Xerial):

Part of #12 (found by this suite's first full run against current main).

Eleven commits hardening the T1/T2 measurement so a verdict can be trusted:
every one exists because a real run produced a number that was true about the
wrong thing.

What changed

  • Arrival is a box, not a shaftrun.arrive_z bounds height; dm3 puts
    walkable floor 344 units inside several targets' own squares.
  • Loadout is declared — every attempt starts from a stated
    health/rockets, because what the bot carries decides which routes exist;
    and rockets double as the permission: the owner's rule is that rocket
    jumping is legal on exactly one route (the pent jump), so an attempt that
    jumps anywhere else is rocketjump — void, not slow.
  • A drill the build cannot be asked is withheld, not failed
    [requires] names the capability and its witness cvar; absence declares
    the drill in capabilities.unavailable instead of grading the map's gap
    as the bot's failure.
  • Routes are gated[route].via carries ordered waypoints lifted from
    the owner's own recorded runs (never derived from geometry); arriving off
    them is offroute. One waypoint carries a measured per-point width, with
    the reason beside it.
  • An attempt we cut short is abandoned, not timeout — fired when the
    speed ceiling proves arrival impossible before the deadline; carries the
    bound it could not beat.
  • The graph is verified before the bot that walks itnav_preflight
    waits for a ready navmesh on the right map and stamps
    cells/links/rj_links/waited_s into the envelope (T1/T2 only: the stamp is
    singular and the two-sided tiers have two graphs).
  • T2 keeps each stall's LinkKind next to its watchdog (kinds per zone,
    offroute naming the empty kind) — a jump-pricing change was invisible to
    the tier built to watch the bot roam, and the probe that measured one had
    to be written outside the suite.
  • quick_attempts lets a drill pin its own quick count (never above
    full); ring_to_ratop pins 5 — its verdict on 3 attempts flipped between
    runs, and on 5 it passes 5/5.
  • Teleport always carries a velocity on the wire — current main's
    Cmd::Teleport requires vel (see bot/steer: unguarded route slice panics and cores the server on dm3 (main @ 9702a1b) #12 discussion); zero is the engine's
    documented plain placement, and pre-field engines ignore the unknown key
    (verified against both).

Schema, validator, fixtures (14 valid / 33 broken), dashboard and docs move
together in each commit.

Evidence

  • Full five-tier column against 817849a: T0 690/690 · T1 4 PASS/16 FAIL/1
    withheld · T2 MEASURED · T3 PIPELINE-OK (+9) · T4 runs.
  • Full five-tier column against current main + the bot/steer: unguarded route slice panics and cores the server on dm3 (main @ 9702a1b) #12 fix: all tiers
    complete, T3 PIPELINE-OK (+11), T4 ladder runs to completion. No drill
    changed verdict between the columns; the status mix and timings did, which
    is the point of the instrument.
  • This suite's first run against current main is what surfaced bot/steer: unguarded route slice panics and cores the server on dm3 (main @ 9702a1b) #12 (server
    core), the same panic in the client seat, and the Teleport.vel wire
    break — all reported there.
  • Offline gates green throughout: schema selftest, dashboard selftest,
    generator regeneration byte-identical.
  • Independent validation of the ruler itself (before the upstream sync):
    every verdict of a full run read back off the recorded demo it was measured
    on — 0 contradictions across 129 attempts; every verdict change between old
    and new ruler attributed to a named rule over the same recording.

Draft until an independent cross-model review has run per our convention.

Posted by Claude via @Xerialen.

@Xerialen

Copy link
Copy Markdown
Contributor Author

Claude (on behalf of Xerial):

Independent review record. Reviewer: a separate Claude session on a different
model tier (Sonnet, high thoroughness), read-only access, instructed to try to
sink the PR rather than confirm it, and to verify every claim itself — it ran
all four offline gates on the live checkout and reproduced its findings by
mutation before reporting them.

Findings, and what was done about each (fixed in 2fe1b8c)

Blocker — the quick rule contradicted the pin feature. checks.py still
required exactly three attempts per drill on a quick run; this branch's own
last drill-fix pins ring_to_ratop to five under --quick. The first quick
run to reach that drill would have had its complete envelope demoted to
failed at write time, aborting the tier. The reviewer proved it by mutating
a valid fixture. Root cause of the escape: no fixture exercised the quick
regime at all. Fixed: the rule keeps its floor (never fewer than three) and
gives up the exact count, which stopped being checkable when pins moved into
scenario files the envelope deliberately does not embed; t1_quick_pinned.json
(surgery on a real quick envelope, validated before becoming a fixture) and
t1_quick_two_attempts.json now hold both directions, and a live --quick
run against the current column completes with ring_to_ratop on five attempts.

Should-fix — a null bound validated. abandoned with min_possible_s: null passed; only the absent key was rejected. Now rejected with its own
message and fixture (t1_abandoned_null_bound.json).

Nit — rejections were not pinned to their reasons. The selftest accepted
any ValidationError from a broken fixture, so a validator edit could make a
fixture fail on an earlier, unrelated check and stay green. The reviewer
hand-verified all current fixtures fail as their names promise; that
verification is now mechanical: schema/fixtures/broken/expected.json pins
every fixture to a fragment of its promised message, and failing for the
wrong reason fails the suite.

What the reviewer examined and found sound

_outcome's via-chain matching, abandon bound, give-up window, arrive box
and rocketjump void (confirming exactly one scenario grants rockets — the
one-legal-route rule); the nav-block gating and both T2 invariants, wired to
real data; the scenario schema in isolation; the teleport wire encoding
(msgpack named fields, engine derives without deny_unknown_fields,
consistent with the live-verified tolerance claim); normalize_nav's
degradation; and that verify_against_evidence.py genuinely compares the
built page against independently loaded envelopes rather than itself. One
declared trust boundary: waypoint provenance was checked to the level of the
eighteen owner demos existing and matching the generator's routes, not
re-extracted coordinate by coordinate.

Gates after the fixes: selftest 15 valid / 35 broken (all pinned), dashboard
selftest PASS, generator regeneration byte-identical, dashboard readback 120
fields / 0 discrepancies against the latest unbroken five-tier column.

Posted by Claude via @Xerialen.

@Xerialen
Xerialen marked this pull request as ready for review July 30, 2026 18:21
Xerialen and others added 14 commits July 31, 2026 09:37
Half the drills on dm3 have walkable ground on another floor inside their
own arrive square — the RA targets have floor 344 units below them — because
the box was tested on X and Y only. A full T1 run put the bot in that square,
on that floor, twelve times: ra_climb five, hex_ssg_to_ratop five,
spawn_sngspawn_to_ring_to_ratop twice, all between 320 and 344 units down.
None was credited, but nothing here prevented it. What stood in the way was
the engine's `arrived` not landing in the same instant, plus one drill's own
fall gate.

Arrival now requires the height too, through `run.arrive_z` (default 48,
zero restores the old behaviour for a drill that asks about a place rather
than a floor). Giving up deliberately does not: both give-up tests keep
asking the height-blind `inside_column`, because a bot at the right place
and the wrong height is still trying. The impossibility bound already had
that exemption and said why; the wedge test did not, so a bot standing on
the target could be declared wedged. It has it now.

Measured on the rig against the same six drills before and after: same six
verdicts. sng_mega is the case the tolerance had to clear — it is credited
24 units above its declared target on every attempt and stays PASS 10/10.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
What the bot carries decides what the planner will consider: a rocket jump is
priced away for a bot that cannot fly one. Left to the map that is whatever it
picked up earlier in the run, and it moves the answer a long way. Measured on
one drill, same rig, same build, with a repeat arm to rule out drift:

  100 rockets   8.68-11.50 s   54-262 rocket-jump links in the route, fires
    0 rockets  14.27-25.55 s   no links, the phase never leaves Idle
  100 again     9.22-13.64 s   fires again

That drill was the one number in T1 that matched the owner's own time (0.99x).
It matched it by taking a jump the rules do not allow on that route, and it
had the rockets to do so because it had been running for twenty minutes.

Every attempt now starts from a stated loadout via `Cmd::Prep` — which the
engine has had since the jump work, and which was unreachable only because
`_parse_verb` in our own client did not know the verb. Its refusal read like
the engine's. `run.prep_rockets` doubles as the permission: a drill handed
none is a drill where the jump is not sanctioned, which is every route on dm3
except the pent jump.

Starting empty is not staying empty — dm3 hands out rocket boxes. `rj_phase`
leaves `Idle` the moment the bot commits to a jump, a field this loop already
fetched on every poll and nothing had ever read. An attempt that jumps anyway
is `rocketjump`: not an arrival, not a failure to arrive, but void, because it
answered a different question. Verified by handing a bot rockets two seconds
into a drill that was given none — 3/3 void, against slow/timeout without.

Measured after: spawn_sngspawn_to_ring_to_ratop falls from 2 passes at 8.78 s
to slow at 14.66 s, spawn_rl_to_ratop_xer loses its only arrival, and sng_mega
is unchanged at 10/10 — its one rocket had been its slowest attempt.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
rj_pent_to_lifts_to_window_to_quad times out 5/5 on main, unchanged when the
launcher is handed 100 rockets. The route runs through a rocket-jump link out
of the pent lift, and that link is planted by the dm3 navpatch — it is not in
the base graph. Main has no navpatch at all: no patch.rs, no data/navpatches/,
no rtx_rj_cost_scale. So the drill was reporting the absence of a route as a
bot that could not walk one, and it was one of the FAILs in T1's column.

A drill now names what its route needs and how to tell whether the build has
it. The witness is a cvar that ships with the capability, read off the engine
binary rather than asked of the server, for the reason engine_declares already
documents: the control layer's Get answers for names no build ever registered.
Absence is the direction that probe establishes reliably and the only one that
changes anything here — present and unknown both run the drill, and which of
the two it was is recorded rather than acted on. Withholding a drill because
the binary could not be read would turn a rig problem into a silence about the
bot.

The capability is named explicitly rather than derived from a route that turns
out to have no links. A missing capability and a bot that cannot use one it has
are different findings, and only the first is the harness's fault.

A withheld drill carries no verdict, no attempts and no times, counts toward
neither the level's verdict nor the dashboard's denominator, and its name goes
into capabilities.unavailable as `t1:<name>` — the same channel the missing
telemetry uses, because it is the same statement. The two have to agree, and
the validator enforces both directions: a drill withheld in silence would leave
the column reading `5/8 drillar` with nothing to say the eighth was never
asked, and a declaration naming a drill that ran would explain away a number
the run produced.

Also fixes something the loadout commit left behind. generate_from_routes.py
rewrites its drills whole, so regenerating would have silently wiped the
prep_rockets that commit hand-added — and would have wiped this requirement
too. Anything the route manifest does not describe now lives in the generator's
ROUTE_RUN and ROUTE_REQUIRES tables; regeneration is a zero diff. The same
commit left `rocketjump` undocumented in SCHEMA.md and the loadout fields
undocumented in the README; both are caught up here.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
rj_pent_window and rj_pent_to_lifts_to_window_to_quad are the same route:
start [958, 788, -296] against [957.8, 787.8, -296.0], target [978, 302, 56]
against [978.5, 301.5, 56.0]. One of them carries the owner's time and grades
against it. The other has required = 8 and no max_time_s at all, so arriving
is the whole test — and the bot passes it 3/3 in 18.9 s by walking, on a build
where the jump the drill is named after does not exist. It was one of T1's
five PASSes.

Its own comment says the owner's run "took 8.7 s end to end". That is the
length of the demo, not the time on the route: 10.689 minus 2.026. His actual
time is 5.42 s, which is what the surviving drill is anchored on. A limit
derived from 8.7 would have been half again too generous, which is how an
untimed drill turns into a wrongly timed one.

Removing it leaves dm3 with no drill that exercises the pent jump on a build
without the navpatch, which is correct: there is no such route on that build,
and the surviving drill now says so rather than walking around it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
spawn_lift_to_pent_to_pentmega passed every attempt at roughly half the owner's
own time by stepping off the pent ledge and freefalling most of the descent.
The endpoint was right and the run was worthless. Every goto drill had that
hole, not a chosen few: a drill said where the bot ended up and nothing about
how it got there.

Drills now carry ordered waypoints the attempt has to pass through before an
arrival counts. The waypoints are points the owner actually occupied, read out
of his own demos of each route — never derived from the navmesh or from
geometry, because a gate built from the thing it is meant to gate gates
nothing. An arrival with waypoints still unmet is `offroute`: it reached the
target, so it is not a failure to arrive; it did not take the route, so it is
not an arrival either. Void, the same shape as `rocketjump`.

The gates do not replace fail.fall_gate or fail.crossing. Those end an attempt
early and give it an honest name of its own; the waypoints decide whether an
arrival counts. Eighteen of the twenty-three drills have gates — the other five
have no demo of the owner running them, and inventing waypoints for those would
be asserting a route nobody has walked.

Three things this had to survive, none of which a passing test suite would have
caught:

The waypoints advance greedily within one poll. The loop looks at the bot
roughly fourteen times a second and a bot at full speed covers sixty units
between samples, so one sample can sit inside two consecutive waypoints.
Advancing one per poll would have called a bot that took the route exactly
`offroute`.

The gates were checked at the rate the runner actually polls, not the rate the
demos sample. His demos carry a point every 0.02 s and the runner sees roughly
every 0.07 s; a gate that only holds at demo density would have failed every
route at once, live, and read as the bot having stopped taking the routes. All
eighteen hold down to 0.16 s.

The box is one width for every waypoint, and 96 is a measured choice rather
than a tidy one. Per-route widths derived from a straight-line shortcut came
out as small as 32, and at 32 the gate called a bot that missed by 47 units
off-route — that is movement, not a shortcut. The straight line turned out to
be a weak measure: the owner's waypoints sit so far off it that anything up to
256 passes. What pins the number is the band where all three measures hold
steady at once — 80 to 112 — and 96 sits in the middle of it, comfortably above
the sixty units a poll can miss.

Replayed against the recorded run on main, five drills change and every one of
them is a bot that went somewhere else: 200 to 717 units off the waypoint it
skipped. spawn_lift_to_pent_to_pentmega goes from three passes to three void
attempts, which was the finding that started this.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
`timeout` covered three different statements: the clock running out, the wedge
test ending an attempt that had stopped moving, and the impossibility bound
cutting short an attempt that was still travelling the moment arriving *in
time* became unreachable. The third is the weakest claim of the three and it
read in the column as the strongest — a bot that never got there.

Across the whole evidence archive, 444 attempts are recorded as `timeout` and
415 of them are that third case. Of those, 182 could still have arrived within
one second of the limit. "The bot cannot walk this route" and "the bot is a
second too slow" are different findings, and until now they were the same
symbol on the page.

So the bound's own outcome is `abandoned`, and it carries `min_possible_s`, the
time it could not have beaten. The validator now requires the two together in
both directions: an abandoned attempt without its bound has thrown away the one
thing it knew, and a bound on any other status is a number nothing produced.
The dashboard puts the bound on the face of the cell — `>=8.4s` — rather than
in a tooltip, because that number is the entire difference between this and a
timeout.

The give-up logic itself is untouched. It works as specified; this only stops
it from lying about what it did. The wedge case keeps the plain `timeout` and
keeps the trade that comes with it: an attempt cut there is still
indistinguishable from one that would never have arrived, and `give_up_grace_s`
is still how you buy that back.

`abandoned` is deliberately not grouped with `rocketjump` and `offroute`. Those
are void — they answered a different question. This one answered the right
question and we stopped listening, so it stays a failure to arrive and counts
as one.

Two consequences worth stating rather than discovering later. Existing evidence
on disk no longer satisfies the tightened rule, since it pairs `timeout` with a
bound; nothing revalidates those files — the dashboard reads the contract
without importing the validator and the sweep reads raw JSON — so they keep
rendering, but a fresh validation of an old envelope would now reject it. And
nine fixtures used that same pairing as filler, so their filler moved to
`abandoned`; each broken fixture was re-checked to confirm it still fails on its
own intended bug and not on the new rule.

Also fixes a fixture that had quietly stopped testing its own name:
`scenario_bad_major.toml` was rejected for missing `category` and `place`,
fields added to the schema after it was written, so it never reached the
major-version check it exists to exercise.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
The suite measured navigation without ever checking there was a navmesh to
navigate. The control layer has answered this all along — status carries
navmesh, cells, links and rj_links — and T1 read status exactly once, at
connect, and used it only for the engine digest.

What the rig actually does, measured rather than assumed: a control port with
no bots requested reports navmesh "none" with all three counts at zero,
because rtx-game builds the graph lazily — bot/population.rs calls
ensure_navmesh only once bots are wanted. Asking for a bot is what starts the
build. On dm3 that build takes 19 s, and for all 19 of them the server answers
with zeros; the first bot appears in the same instant the graph does, because
a bot cannot spawn without one.

So the old code worked by accident. The 40 s bot wait covered the 19 s build
with room to spare, and when it did not, it said "server did not expose 1 live
bot(s)" — a true sentence about the wrong thing. Nothing anywhere recorded
which graph the numbers had been measured against, so two runs against
different map knowledge were indistinguishable afterwards.

nav_preflight polls status about once a second until the graph is ready and
the map is the one we asked for, and only the deadline is a verdict.
Everything short of ready is "not yet": "building" is the obvious case, and
"none" means no build is in flight yet, not that none ever will be. Reading
status once and judging the answer is the trap the whole function exists to
avoid, and the zeros it returns are exactly what a broken build would look
like. The wait is 120 s, twice what the engine's own MCP helper gives the
identical condition, because a lab rig sharing a box is slower than the laptop
that number was tuned on. It sits after the bot count is set and before the
bot is waited for, which is the only placement that works and also the one
that makes a failure legible.

The result is stamped into the envelope as nav, beside build: the map, the
state, the three counts, and how long the poll waited. cells and links must be
positive — a ready graph with zero cells is not a graph, and that is the
concrete case the "a value that could not be measured is null, never 0" rule
exists for. rj_links may be zero, because a build with no rocket-jump links is
a legitimate build. waited_s is provenance rather than measurement: it says
whether the rig was hot or cold when the numbers were taken.

The block is required on every complete T1 or T2 envelope and rejected on
every other tier. Not because the others never connect — T3 and T4 both hold
control connections — but because the stamp is singular and they are
two-sided: each side runs its own client build with its own graph, and one
block beside one build could not say which side it described. Rather than
stamp an ambiguous graph they stamp none.

T1's dash is gated but deliberately not stamped. It runs on its own map after
a map change, and that change drops the graph entirely — the fixed 5 s settle
there was a guess about a background build that takes four times as long. The
preflight is the floor under that guess. A second graph identity inside an
envelope whose map field names one map would only invite the reader to
attribute one to the other.

Two consequences worth stating rather than discovering. Every T1 and T2
envelope already on disk now fails the tightened rule, since none of them
carry a stamp; nothing revalidates those files, so they keep rendering, and
the dashboard shows old runs exactly as before. And every existing T1/T2
fixture had to gain a block — a broken fixture without one would have started
failing on the missing stamp instead of on the bug it was written to catch,
with the suite green and testing nothing. The rejection message of all 26
broken fixtures was captured before the change and diffed after: all 26 are
byte-identical, and the five new ones each fail for the reason their name
promises.

Measured on main, on the rig: the run stamps cells 4634, links 36956,
rj_links 2021, waited 0.02 s on a graph left warm by the previous run, and
comes out at 5 of 22 drills with one withheld. Against the last recorded run
exactly two drills differ, and both were already known — the untimed pent
duplicate that this branch deleted, and spawn_lift_to_pent_to_pentmega, which
the route gates turned from three passes into three void attempts.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
Eighteen of the twenty-three drills got waypoints from the owner's own demos.
The remaining five had none, and the reason given was that he had no recording
of them — which was true of the demo *files* and false of the routes. He
resolved all five in one pass, and searching his eighteen demos on his answers
turned up what the first search had missed: a drill can be a leg of a longer
route he ran, not only a run of its own.

hexagon_sod_tur is dropped at his instruction. Worth recording because nothing
else would have: it was the only drill that planted runtime links, and those
four links stayed in the graph for every drill that ran after it in name order
— a dozen of them, none of which asked for them. That coupling is gone with it.

ra_climb is dropped as a duplicate. He asked whether there was a
ra-bottom-to-ra-top demo; there is, and it already anchors ralow_to_ratop with
his own 7.78 s and an 8.71 s limit. ra_climb was the same climb to the same
platform — the targets sit twenty-seven units apart — from a bottom he never
ran, and graded on nothing but "ten arrivals of ten". That is the pair
rj_pent_window formed with its own twin, and it ends the same way.

The two cell probes keep no waypoints, and the exemption is now written in
their files instead of being an unexplained blank. A cell probe *is* its pair
of cells; there is no path to take a different version of, and a gate built
from the cells would be built out of the thing it exists to check.

sng_mega is gated, and so his statement that a demo already existed for it was
correct. Two of his runs cover the leg; the waypoints come from the one that
passes the drill's own start closest, twenty units off it and twenty-five off
the target. Both were checked the way the eighteen were: his path clears the
gate down to a sixth of the sampling rate the runner has, and a straight line
from start to target does not.

Twenty-one goto drills remain, nineteen gated, and the only two without a gate
say why in their own text.

Measured on main after the change: three of twenty graded, one withheld.
Against the run before it, exactly one drill moved that was not deleted —
ring_to_ratop, from pass to fail. That one is not yet attributable: it is one
of the drills that has been flapping between a pass and a slow arrival all
along, and it is also the first drill in name order to lose the four planted
links described above. One run cannot separate those two, and this commit does
not claim to.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
T2 already keeps `reasons` per zone — which watchdog fired — but dropped the
LinkKind of the route leg the bot held when it did. Those are different
questions: `air_commit_off` says a jump was committed and missed; only the kind
says it was a JumpGap rather than a SpeedJump runway.

The measurement that earned this field its place: pricing jump links by
envelope share cut JumpGap stalls 296 -> 57 across three ten-minute runs per
column, and that number could not be produced by the suite. The probe had to be
written by hand, outside it, because the envelope threw the kind away at the
door. A pricing change on one link kind was invisible to the very tier built
to watch the bot roam.

Each zone now carries `kinds` next to `reasons`, with "offroute" naming the
empty kind the engine sends when the bot held no leg — a name, because an
empty-string key reads as a serializer accident rather than a state. The
writer-side invariant extends to it: firings == sum(reasons) == sum(kinds),
with its own message and its own broken fixture.

The dashboard is deliberately untouched: normalize_cells whitelists its
fields, so the new one flows through the envelope without rendering. Drawing
it belongs with the map view, not squeezed into this commit.

Old T2 envelopes on disk no longer validate. Nothing revalidates them, so they
render as before — the same stance every schema-tightening commit on this
branch has taken.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
…-cut gate

Two findings from reading the suite's verdicts back off the demos they were
measured on, each fixed where it belongs.

ring_to_ratop's passing times sit in two bands — 7.50-7.91 and 8.09-8.19
against a limit of 8.42 — and it abandons intermittently, 2-3 of 15 in both
arms of the planted-links A/B (which also showed the links explain nothing:
12/15 with, 11/15 without). Three attempts with a required of two turns that
variance into a verdict that flips between runs, and it has, all along. Drills
may now pin their own quick count, and this one pins five: the quick cut
exists to save rig time, not to grade a drill on less evidence than its
verdict needs. Quick stays a cut — the schema rejects a quick count above the
full one.

spawn_rl_to_ratop_xer's first waypoint gets its own width. The bot corner-cuts
'bridge låg' at 97.4-103.7 units on the worst axis in all six recorded
attempts while clearing the three later waypoints by 13-26 — and the ordered
chain means a first-waypoint miss silences the rest. It has never changed a
verdict (those attempts all fell to stall, rocketjump or the abandon bound
first), but a gate that would void a route-following bot by one unit is a trap
armed. 128 covers the corner-cut; the straight line passes 311 from this
waypoint, so the gate keeps a 2.4x rejection margin, and the owner's own run
goes through at 0.1. Via entries can carry a per-waypoint width for exactly
this case, reason in a comment beside the entry.

Both drills are generator-owned, so both changes live in generate_from_routes
and the regenerated files differ by exactly the two intended lines.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
Upstream grew Cmd::Teleport a vel field without a serde default, so an engine
past fed5e36 times out every frame this runner sent — T1 died on its first
teleport while T2, which never places a bot, measured happily on the same
socket. Zero is the engine own documented plain placement, and engines from
before the field ignore unknown fields — verified live against both 817849a
and the new main on the lab rig. Optional trailing coords pass a real
velocity through, for reproducing a moving start rather than a position.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
Everything else in the suite runs deterministically from one command, but two
pieces of today lived outside it. tools/cargo_summary.py turns cargo test
output into the summary t0-import expects — without it, T0 needs a person to
hand-build the JSON, which is the one step an unattended run could not do.
dashboard/verify_against_evidence.py reads the RUNS JSON back out of the built
page and compares it field by field against the raw envelopes: never trust
the pipeline, read the output. It accepts exactly two presentation
transforms — the Swedish verdict labels and the ladder padding its unplayed
rungs — and everything else must match, 120 fields on the current column.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
…, and every broken fixture is held to its promise

Three findings from the independent review of this branch, each fixed with
the fixture that was missing when it slipped through.

The blocker: the validator still said a quick run has exactly three attempts
per drill, written before drills could pin their own quick count — so the
first --quick run to reach ring_to_ratop (pinned to five) would have had its
complete envelope demoted to failed at write time, aborting the tier. The
rule is now the floor alone: quick never grades a drill on fewer than three.
The exact count stopped being checkable the moment pins moved into scenario
files the envelope deliberately does not embed; what remains is `of` equal to
the attempt count (already enforced) and the schema rejecting a pin above the
full count at load. No fixture exercised the quick regime at all, which is
how this survived — t1_quick_pinned.json is surgery on a real quick envelope
from the rig, ring_to_ratop extended to its five-attempt pin with every
dependent field recomputed, and t1_quick_two_attempts.json proves the floor
still fires. Verified live as well: a --quick run against the current column
completes, ring_to_ratop on five attempts, envelope validates.

The hole: an abandoned attempt with min_possible_s present but null passed
validation — only the absent key was rejected. An abandoned attempt exists
because a bound was computed; null is the same thrown-away knowledge wearing
the key. Rejected now, with its own fixture.

The rot guard: the selftest accepted any rejection of a broken fixture, so a
validator edit could make a fixture fail on an earlier, unrelated check and
stay green. schema/fixtures/broken/expected.json pins each fixture to a
fragment of its promised message; a fixture failing for the wrong reason now
fails the suite, and so does a pin without a fixture.

Review: independent session on a different model tier, read-only, ran the
offline gates itself; findings verified by mutation before being reported.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01WsgHA2RQVwyQ9ArJquyNJi
Define measured, complete, green, and published outcomes; require semantic gates for every tier; and document pinning, evidence, retraction, remote verification, and restoration.

Add an independent T2 powerup watcher that rejects dirty preflight state and disagrees loudly with overwritten envelope values. Teach dashboard readback to verify failed tiers, floors, and metric sources.

Co-Authored-By: OpenAI GPT-5.6 <noreply@openai.com>
@dsvensson
dsvensson merged commit 43fba2c into qw-ctf:main Aug 1, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

2 participants