Skip to content

v7-context

@Baradelli Baradelli tagged this 24 Sep 14:47
The fix round was not clean. Four findings from a scoped re-review, plus two
things found while fixing them. No model calls: the one experiment-shaped thing
in this commit is proving that `npm test` *would* have made them, and that was
done with `--runs 0`.

**`npm test` fired a hundred paid runs on any machine with a key.**
`tools/run-context.ts` called `main()` at module scope and
`tools/context/tally.test.ts` imports two constants out of it, so the test
suite started the hundred-run experiment — about $2.26 of billed calls and live
requests to somebody else's encyclopaedia — from the command whose whole promise
is that it is free to run. It looked honest only because the machine it was
written on had no key exported, where it fails fast into a gitignored file.

Shown at zero cost: the old module checked out beside the new one, each
`import`ed rather than executed, with `--runs 0` so the plan is empty.

  --- OLD (unguarded), imported for a constant, --runs 0 ---
  [context] 0 run(s): 2 condition(s) x 0, ceiling 16000 tokens, cache "use"
  [context] wrote <scratch>/probe.tsv
  --- NEW (guarded), same import ---
  (nothing printed: main() never ran)

`run-judge.ts` already had the right shape, and so did seven other commands.
Eight did not, and do now: `run-context`, `run-roster`, `run-summary-judge`,
`window-probe`, `warm-cache`, `make-covers` and `record-trace`, plus
`agent/src/cli.ts` — outside the brief's `tools/*.ts`, but importing it for
anything starts a billed agent run, which is the same hole.

`tools/entrypoints.test.ts` makes it the rule: every non-test file under
`tools/` and `agent/src/` that defines a `main` must carry the guard and must
not call `main(` in column zero. Sixteen files, 33 assertions. It reads source
rather than importing anything, deliberately — a test that proved the guard by
importing the module would be a test that spends money once the guard is gone.
Delete the guard from `run-context.ts` and two assertions go red, naming it.

The predicate was checked under `tsx` before it was trusted, because a guard
that is never true is a command that silently does nothing: it holds from the
repository root, from `agent/` with a relative path, and with `--env-file` in
the way. `npm run covers`, `npm run validate` and `npm run eval` exercise three
of the newly guarded files end to end and still do their work.

**A fifth source still said the cache was committed.**
`tools/context/task.test.ts` said it checked "against the committed cache",
which was wrong twice: `webcache/` is gitignored, and the block that sentence
heads reads the two published traces, not the cache. It now says what the block
checks — the page bodies the model was actually handed — and says it is not the
cache and why. `tools/context/task.ts` made the same claim in different words
("against the cached pages") and is fixed with it; the grep for "committed" does
not find that one, which is the honest lesson about greps.

`git grep -i committed | grep -i cache` at `fe21ff7` returns seven sources: two
true, and **five** false in three files. The previous commit swept four of the
five and its message said "three committed files said it was … All four now say
what is true". Both numbers described the sweep rather than the tree, so that
message is amended to the true count and to naming the one it missed.

**Two comments wider than the code, both left in place by the last round.**

- the doc on `UNTRUSTED_OPEN` still said a reader of a trace can see exactly
  which bytes came from somewhere else. Search titles and snippets are
  third-party bytes and arrive unfenced, which `index.mdx` and `defuseFence`'s
  own doc fourteen lines below already said. Narrowed to match: the fence marks
  page bodies and only page bodies. Search results are still not fenced, on
  purpose — fencing them changes frame 3 of the control trace and moves the
  708-token baseline the 5,644 / 2.12% figure is measured against;
- `MIN_REQUEST_GAP_MS` said "minimum gap between two live requests".
  `politeFetch` writes `lastRequestAt` before the `fetch`, so four concurrent
  callers read the same value, sleep the same amount and fire together. The
  comment is fixed and the behaviour is not: it cannot be exercised without
  fetching from Wikipedia, and untested code in the one path that touches
  somebody else's service is worse than a wrong comment. It now says it bounds
  how often a burst of four starts, not how far apart two requests are.

Two more, found while fixing those:

- `run-context.ts` carried `/** One at a time. … */` over `const CONCURRENCY =
  4`, and has since `fe21ff7`. It is the same constant the floor above turns on;
  the comment now says four and says what that does to the floor;
- `descriptions.ts` discussed v7's two new tool descriptions without noting that
  both are wider than the tools: `web_search` says "the public web" and searches
  one encyclopaedia, `fetch_page` says it only accepts URLs `web_search`
  returned and accepts any `/wiki/<article>` URL. **The strings are unchanged** —
  the model read those exact bytes in all 100 runs and they are inside the
  708-token baseline — but the header says so now.

No published artifact is touched. This commit is comments, entry-point guards
and one test file; `git diff` over `site/src/content/posts/` is empty, and
`npm run covers` leaves all seven covers byte-identical.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Assets 2
Loading