chore(eval): pin all agents to Sonnet 5 and de-duplicate LLM response metrics - #116
Merged
David Koleczek (DavidKoleczek) merged 1 commit intoAug 5, 2026
Conversation
… metrics Pin every agent-under-test to claude-sonnet-5 so cross-agent comparisons differ only by agent stack, not by model. Drop the `routing.matrix: opus48` reference from the amplifier-foundation settings: no such matrix exists in the routing-matrix bundle, so the model was already coming from `default_model` alone. Enable `debug.rawLlmPayloads` for amplifier-agent-local. The full request and response payloads ride on the `llm:request` / `llm:response` events that extraction already pulls, so no extraction change is needed. Fix double-counted metrics. A session that composes more than one logging hook writes each LLM call to disk once per hook, and the metrics pass summed across every extracted events.jsonl. The anchors bundle composes two such hooks, so the amplifier-foundation agent reported exactly double its real calls, tokens and cost. The two copies use different envelope shapes and share no identical lines, so they are de-duplicated by response identity: the provider response id when raw capture is on, otherwise a session/timestamp/usage fingerprint. Events carrying neither are counted rather than dropped, since understating cost is the worse failure. The correction is reported in the metrics notes. Add regression tests for the de-duplication, and a dev dependency group so they run with `uv run python -m pytest tests/`. pytest-asyncio is required by amplifier-core's pytest plugin, which loads via entry point at startup. 🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier) Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Three changes to the evaluation harness in
.amplifier/evaluation/.Pin every agent-under-test to
claude-sonnet-5. Cross-agent comparisons nowdiffer only by agent stack, not by model. This also removes the
routing.matrix: opus48reference from the amplifier-foundation settings — nomatrix by that name exists in the routing-matrix bundle, so the model was
already resolving from
default_modelalone. Naming a real matrix instead wouldhave re-introduced role-based fan-out and defeated the single-model pin.
Enable
debug.rawLlmPayloadsfor amplifier-agent-local. The full request andresponse payloads ride on the
llm:request/llm:responseevents thatextraction already pulls, so no extraction change was needed.
Fix double-counted metrics. A session that composes more than one logging
hook writes each LLM call to disk once per hook, and the metrics pass summed
across every extracted
events.jsonl. The anchors bundle composes two suchhooks, so the amplifier-foundation agent reported exactly double its real calls,
tokens and cost.
The duplicate copies use different envelope shapes and share no identical lines,
so byte comparison cannot detect them. They are now de-duplicated by response
identity: the provider's response id when raw capture is on, otherwise a
session/timestamp/usage fingerprint. Events carrying neither are counted rather
than dropped — understating cost is the worse failure mode. The correction is
reported in the metrics notes rather than applied silently.
Verification
Re-ran the metrics pass over an existing 3-agent x 3-task run. Only the three
affected cells change, and each halves exactly; all other cells are untouched:
tests/test_metrics_dedup.pyadds 8 regression tests built from the two realenvelope shapes, covering: duplicate collapse, distinct calls staying distinct,
de-duplication with raw capture off, unidentifiable events being counted rather
than dropped, the correction appearing in notes, and wallclock being unaffected.
A dev dependency group makes them runnable with
uv run python -m pytest tests/.pytest-asynciois included becauseamplifier-core's pytest plugin loads via entry point at startup and fails
collection without it.
Checks: 8 tests pass, ruff clean, pyright unchanged from baseline,
run.py validatepasses for all 4 agents and all tasks.