A result says which question set it is, and an arm's CLI is on PATH - #11
Open
lex00 wants to merge 3 commits into
Open
A result says which question set it is, and an arm's CLI is on PATH#11lex00 wants to merge 3 commits into
lex00 wants to merge 3 commits into
Conversation
…is on PATH
Two things that stop the negative question set being publishable, and one of
them was invalidating runs.
## The scenario a job scored
`emit-result.py` hardcoded `SCENARIO = "ec2-multiregion"`, so a negatives run —
two questions, six trials — emitted a record claiming to be a run of the
eight-question board. The emitter cannot work it out for itself: `result.json`
names the tasks but not the set they came from, and deriving the set from its
members would make a partial run look like a different scenario.
So the script that ran it writes `jobs/<job>/scenario`, and the emitter reads
it. Absent, it stays `ec2-multiregion`, which is what every job recorded before
this was.
This is not cosmetic. The board is over 24 trials and the negative set is over
6; `validate_results.py` groups by `(bench, scenario)` and rejects a group whose
runs disagree on `expected_trials`, so a mislabelled record does not quietly
join the board — it fails the whole group.
## The arm's CLI, under the name its briefing uses
chant's briefing teaches `chant search …` and alchemy's teaches
`alchemy state list`. Neither was on PATH. The agent's first call died with
`command not found`, and it then spent turns rediscovering `npx <tool>`.
Every trial of a chant run did this, and the audit — correctly — reads a missing
CLI as "this run did not measure the arm" and invalidates the run:
24 of 24 trials could not find chant on PATH — the run is not a measurement
of this arm
TOOLING NOT EXERCISED
That run also showed what the flailing costs: 218 tool invocations against the
67 a clean run takes, and 5 of 24 trials passing.
The arm's `node_modules/.bin` now joins PATH in its image. This is not a thumb
on the scale for the arms that have a local CLI — the briefing is part of the
experiment, and an environment that does not provide the command its own
instruction names is measuring the agent's recovery from a broken setup rather
than the tool. terraform, pulumi and cdk are unaffected: theirs are already on
PATH or invoked through npx.
Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
The first version of this set ENV PATH in the arm image. That image is not what a trial runs: prepare.py exports the workspace and the harness bind-mounts it into a container built from the dataset's own image, so the ENV never reached a trial. prepare.py's own docstring says as much — "a trial's image comes from the dataset, so the prepared image cannot be the container the agent runs in" — and I put it in the one place that cannot work. It goes in the agent env the runner passes instead, which is what actually reaches the trial. The base PATH is read out of the tools image rather than written down, so it cannot drift from the Dockerfile that sets it. Measured rather than assumed this time: chant-r2 ran with the arm-image version in place and was still voided, 24 of 24 trials, at 21/24 with 72 healthy invocations and a 1% failure rate. Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lex00
added a commit
to INTENTIUS/chant-bench
that referenced
this pull request
Aug 3, 2026
The chant arm vendored 0.33.1. It now carries INTENTIUS/chant#1425 and #1429, so its published g-series describes a build the arm no longer runs: different harness commit, different workspace fingerprint. By the rule method.md states, that is a different experiment. Three replicates rather than one. A single new run would have joined a set whose other two members are the old build, and the board ranks on the MEDIAN of three — so the figure would have described neither build. The set is now homogeneous: chant-r3 24/24 workspace 3912387ce569 harness 4f3ab36-dirty chant-r4 22/24 workspace 3912387ce569 harness 4f3ab36-dirty chant-r5 22/24 workspace 3912387ce569 harness 4f3ab36-dirty Median 0.9167, which is exactly what the g-series median was. **The two fixes do not move the board**, and that is the expected result rather than a disappointment: the arm's 0.33.1 read the estate through the CLI, which passed `--region` and signed it, so it already saw all three regions. #1429 is a regression against a build the board never ran. Where they do move the number is the negative set, which asks the question the board does not: 0/6 to 3/6. All three runs answered with zero account reads, entirely from the recorded snapshot, and all three passed the audit — 6/6, 24/24 and 24/24 trials using chant's own tooling, at 3%, 5% and 1% invocation failure rates. Two earlier attempts are not published. Both were voided for the same reason, before lex00/aws-bench#11 put the arm's CLI on PATH where a trial could see it. Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ported since `workspace_fingerprint()` read `<arm>.fingerprint`, which every `prepare.py --export` overwrites. Emit a run after the next export and it is stamped with a workspace it never saw. Silent, and easiest to hit exactly when it matters: several runs ingested together after the rebuild that follows them. Three published runs claimed a build that postdated them — measured before a fix, stamped as though measured after it, which is the difference the field exists to record. Same shape as the `harness_commit` fault in INTENTIUS/chant-bench#26: a provenance field derived at emit time from state that moves, rather than captured when the thing it describes happened. The runner now copies the fingerprint into the job, and the emitter prefers what the run recorded. Older jobs have no such file and fall back to the previous behaviour, so nothing already published changes shape. Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lex00
added a commit
to INTENTIUS/chant-bench
that referenced
this pull request
Aug 3, 2026
The middle of a replicate set rather than a single run, the same rule the board
uses — and here it matters more, not less. Six trials move further than
twenty-four do: on one build these questions returned 3, 4 and 6 of 6 with
nothing changed between the runs.
before the kind fix 3/6, 4/6, 6/6 median 4/6
after 6/6, 6/6, 6/6 median 6/6, no spread at all
That is a real shift rather than another draw from the same distribution, and
the variance collapsing is the more convincing half.
## What was making it noisy
Not the traversal — the counting. `kind:` is a substring, and a substring of a
CloudFormation type crosses kind boundaries: `kind:EC2::VPC` also matched
`AWS::EC2::VPCGatewayAttachment` and `kind:EC2::Subnet` also matched
`AWS::EC2::SubnetRouteTableAssociation`. An estate holding 6 VPCs answered 9.
Agents were naming exactly the right VPCs and reporting "9 VPCs in the estate"
beside them, and the grader failed the answer for contradicting the estate —
correctly, because it did. The finding was right and the census next to it was
of a set nobody asked for. Fixed in INTENTIUS/chant#1451.
The traversal work that made the questions answerable at all is
INTENTIUS/chant#1452, which replaced a per-rule opt-in with containment being
traversable by construction.
## A provenance error, found and corrected
`chant-neg-3/4/5` were published claiming workspace `30c405b9a8f5`. They ran
against `787ab0fd5bd0`; the emitter read the fingerprint file at emit time and
that file had been overwritten by a later export. Corrected here from the build
logs, and made impossible in lex00/aws-bench#11 — the runner now records the
fingerprint into the job.
Worth stating plainly because the correction changes what the table says: the
three runs either side of the kind fix are NOT the same experiment, and the
mis-stamp had made them look like they were.
Refs #27
Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
lex00
added a commit
to INTENTIUS/chant-bench
that referenced
this pull request
Aug 4, 2026
…ll I built
The arm vendored a packed build. Every figure until now was measured against a
tarball produced on this machine, which is a different claim from "the released
package does this" — the code being right and what ships being right are not the
same fact, and only one of them is reproducible by a reader.
The arm is pinned to published `@intentius/chant` 0.41.0 and
`@intentius/chant-lexicon-aws` 0.41.0. Nothing local remains in its dependency
tree, so `npm install` reproduces these runs.
## Both sets re-measured on the published package
board 23/24, 23/24, 23/24 median 0.9583 (was 0.9167)
negatives 6/6, 6/6, 6/6 median 6/6 (was 0/6 at the start)
Both sets homogeneous — one workspace fingerprint, one harness commit, one
briefing SHA — and both with no spread at all. Every run answered with ZERO
account reads, entirely from its recorded snapshot, and every run passed the
audit.
The negatives figure reproduces the local build's exactly, which is what this
re-run existed to check. The board moved up rather than staying still; the
published package carries more than my six fixes (the CloudFormation spec
upgrade among them), so that difference is not attributed here.
## Provenance is now recorded rather than inferred
These carry workspace `09766f0396b7`, distinct from the local build's
`30c405b9a8f5`, because lex00/aws-bench#11 makes the runner stamp the
fingerprint into the job at run time. Under the previous behaviour — read from
the export at emit time — ingesting these after any later rebuild would have
labelled them with a workspace they never ran against, which is exactly the
error that had to be corrected by hand two commits ago.
Refs #27
Claude-Session: https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two things standing between the negative question set and a published number, one of which was invalidating runs outright.
A result says which question set it is
emit-result.pyhardcodedSCENARIO = "ec2-multiregion", so a negatives run — two questions, six trials — emitted a record claiming to be a run of the eight-question board.The emitter cannot work this out for itself.
result.jsonnames the tasks but not the set they came from, and deriving a set from its members would make a partial run look like a different scenario. So the script that ran it stampsjobs/<job>/scenarioand the emitter reads that. Absent, it staysec2-multiregion— which is what every job recorded before this was, so the 45 existing ones emit exactly as they did.Worth saying why this is load-bearing rather than tidy:
validate_results.pygroups runs by(bench, scenario)and fails a group whose runs disagree onexpected_trials. A negatives record emitted under the board's name does not quietly land beside 24-trial runs with a denominator of 6 — it breaks the whole group. That check is right and should stay; what was missing is a second scenario for such a record to be valid under.An arm's CLI, under the name its own briefing uses
chant's briefing teaches
chant search …. alchemy's teachesalchemy state list. Neither was on PATH, so the agent's first call died withcommand not foundand it spent turns rediscoveringnpx <tool>.Every trial of a chant run did this, and the audit — correctly — treats a missing CLI as disqualifying:
That run also shows what the recovery costs: 218 tool invocations where a clean run takes 67, and 5 of 24 trials passing.
The arm's
node_modules/.binnow joins PATH in its image, indockerfile_for, so it is one rule for every arm rather than a per-arm patch.This is not a thumb on the scale. The briefing is part of the experiment. An environment that does not provide the command its own instruction names is measuring the agent's recovery from a broken setup, not the tool. terraform, pulumi and cdk are unaffected — theirs are already on PATH or invoked through
npx, which is why this never showed up as a cross-arm anomaly the way the alchemy v2 entrypoint did.Checked
scenariofile emitsec2-multiregion, unchanged.command -v chantin the rebuilt image resolves to/workspace/chant/node_modules/.bin/chant.🤖 Generated with Claude Code
https://claude.ai/code/session_014KoduiMJRyLqjFYBUbMskE