* feat(multishot): freeze the loop's behaviour as golden records
The loop-to-graph parity claim is proven and merged. What keeps value is
regression detection on the graph engine, and that needs the loop's recorded
outputs, not a second live orchestrator.
`@tangle-network/agent-eval/multishot/golden` ships:
- 13 deterministic shot scenarios and 1 matrix scenario. Scripted transports,
scripted executors, fixed personas and token budgets. No network, no clock in
a recorded field, no random number.
- A record per scenario holding the full request ledger of both legs (model,
temperature, token budget, advertised tools, tools-by-reference, message log)
and the outcome: the result without wall-clock duration, or the throw reduced
to its class, message and declared cell spend.
- `assertMultishotGoldenScenario` / `checkMultishotGolden` and the matrix pair,
which throw or report every field that moved. No test framework needed.
- `scripts/record-multishot-golden.ts`, which never picks an engine for you,
refuses to overwrite a released version, and captures every scenario twice so
an unreproducible scenario fails instead of freezing a coin flip.
v1 is captured from `./multishot`'s loop at 0.145.21 — the engine the merged
parity proofs compared against.
The records are load-bearing: a single-field mutation of any recorded leaf,
across all 14 records, is reported as a named mismatch.
Docs: docs/multishot-golden-records.md
* chore(multishot): waive the golden judge fixture id and repin the analyst digest
The golden matrix scenario answers its judge leg from a score table on its own
fetch wire, so `test/judge-model` never reaches a provider and has no served id
to assert. The analyst dependency lock covers package.json, which now declares
the ./multishot/golden subpath.
* fix(multishot): compare advertised tools by value, not by array identity
The request ledger pinned `req.tools === options.tools`. Array identity is not
observable behaviour: an engine that deep-copies the tools before dispatch
offers the agent exactly the same thing, and would have failed the check for a
reason no caller can see. Both review lenses named it as the field most likely
to force a conforming engine to fake an implementation detail.
The ledger now records the tool DEFINITIONS and compares them by value. A
rebuilt array passes; a changed name, description or parameter schema is a
mismatch. v1 is re-recorded against the same loop.
Also documented: the matrix judge wire is process-wide, so matrix checks run
serially per process; and when an old record version may be dropped.
* fix(multishot): refuse a second concurrent matrix judge wire
The wire holds globalThis.fetch for the run, so two matrix checks at once in
one process would cross their judge ledgers and report a mismatch nobody can
explain. The second install now throws instead. The serial rule is enforced,
not only written down.
* fix(multishot): close the golden harness and recorder findings
From the multi-shot audit. The medium finding was a real silent pass; the rest
harden the freeze invariant the module exists to hold.
- `checkMultishotGolden({ only })` reported ok for a run of ZERO scenarios when
an id named nothing. A stale id after a rename now stops the check.
- A durationMs outside the contract returned alone and hid every other
divergence in the same run. It joins the mismatch list instead.
- The recorder now refuses a --version that is not a bare file name, refuses a
flag where a value belongs, refuses the network during a shot capture, and
refuses to freeze a result whose durationMs the check would reject.
- The freeze guard was check-then-write across the whole capture run. The record
is written to a sibling and linked into place, so a concurrent recorder gets
EEXIST and a crash leaves a `.partial` nobody mistakes for a released record.
- The summary-duration mask reads fractional and millisecond renderings, and
fails loud on a duration it cannot mask rather than leaving wall clock in the
comparison.
- Dropped the `record:multishot-golden` script: its three required arguments
have no defaults by design, so the bare script could only fail. The doc
carries the canonical invocation.
- Docs name what `checkMultishotGolden` covers and the registration step a new
version needs.
* chore(release): 0.146.0 — freeze the multishot loop's behaviour as golden records
* fix(multishot): enforce the golden freeze and close the remaining audit findings
All low, none blocking; these are the ones that harden the claim the module
makes about itself.
- The record set is validated on load and deep-frozen. It was handed to every
caller by reference, so "frozen" was a convention rather than a property, and
a malformed fixture would have surfaced as a mismatch on every scenario
instead of naming where it broke.
- The judge wire answers its own endpoint, not every URL sharing its prefix.
- `readRunDir` says which run directory is missing instead of raising ENOENT.
- `compareJson` honours its cap exactly; one frame could push past it.
- The recorder guards `--out` like `--version` and takes a per-process temp
path, so two recorders cannot meet on one `.partial`.
- The doc names `recordedAt` as provenance nothing compares, states the
engine-naming rule without claiming a state this commit has not reached, and
is linked from the doc index.