Skip to content

chore(eval): pin all agents to Sonnet 5 and de-duplicate LLM response metrics - #116

Merged
David Koleczek (DavidKoleczek) merged 1 commit into
mainfrom
eval/sonnet-5-pin-and-metrics-dedup
Aug 5, 2026
Merged

chore(eval): pin all agents to Sonnet 5 and de-duplicate LLM response metrics#116
David Koleczek (DavidKoleczek) merged 1 commit into
mainfrom
eval/sonnet-5-pin-and-metrics-dedup

Conversation

@DavidKoleczek

Copy link
Copy Markdown
Collaborator

Summary

Three changes to the evaluation harness in .amplifier/evaluation/.

Pin every agent-under-test to claude-sonnet-5. Cross-agent comparisons now
differ only by agent stack, not by model. This also removes the
routing.matrix: opus48 reference from the amplifier-foundation settings — no
matrix by that name exists in the routing-matrix bundle, so the model was
already resolving from default_model alone. Naming a real matrix instead would
have re-introduced role-based fan-out and defeated the single-model pin.

Enable debug.rawLlmPayloads for amplifier-agent-local. The full request and
response payloads ride on the llm:request / llm:response events that
extraction already pulls, so no extraction change was needed.

Fix double-counted metrics. A session that composes more than one logging
hook writes each LLM call to disk once per hook, and the metrics pass summed
across every extracted events.jsonl. The anchors bundle composes two such
hooks, so the amplifier-foundation agent reported exactly double its real calls,
tokens and cost.

The duplicate copies use different envelope shapes and share no identical lines,
so byte comparison cannot detect them. They are now de-duplicated by response
identity: the provider's response id when raw capture is on, otherwise a
session/timestamp/usage fingerprint. Events carrying neither are counted rather
than dropped — understating cost is the worse failure mode. The correction is
reported in the metrics notes rather than applied silently.

Verification

Re-ran the metrics pass over an existing 3-agent x 3-task run. Only the three
affected cells change, and each halves exactly; all other cells are untouched:

amplifier-agent-local   (3 cells)   unchanged
opencode-vanilla        (3 cells)   unchanged
amplifier-foundation    finance      20 -> 10 calls   $2.98 -> $1.49
amplifier-foundation    operations   30 -> 15 calls   $8.41 -> $4.20
amplifier-foundation    sales        34 -> 17 calls   $8.57 -> $4.28

tests/test_metrics_dedup.py adds 8 regression tests built from the two real
envelope shapes, covering: duplicate collapse, distinct calls staying distinct,
de-duplication with raw capture off, unidentifiable events being counted rather
than dropped, the correction appearing in notes, and wallclock being unaffected.

A dev dependency group makes them runnable with
uv run python -m pytest tests/. pytest-asyncio is included because
amplifier-core's pytest plugin loads via entry point at startup and fails
collection without it.

Checks: 8 tests pass, ruff clean, pyright unchanged from baseline,
run.py validate passes for all 4 agents and all tasks.

… metrics

Pin every agent-under-test to claude-sonnet-5 so cross-agent comparisons
differ only by agent stack, not by model. Drop the `routing.matrix: opus48`
reference from the amplifier-foundation settings: no such matrix exists in the
routing-matrix bundle, so the model was already coming from `default_model`
alone.

Enable `debug.rawLlmPayloads` for amplifier-agent-local. The full request and
response payloads ride on the `llm:request` / `llm:response` events that
extraction already pulls, so no extraction change is needed.

Fix double-counted metrics. A session that composes more than one logging hook
writes each LLM call to disk once per hook, and the metrics pass summed across
every extracted events.jsonl. The anchors bundle composes two such hooks, so
the amplifier-foundation agent reported exactly double its real calls, tokens
and cost. The two copies use different envelope shapes and share no identical
lines, so they are de-duplicated by response identity: the provider response id
when raw capture is on, otherwise a session/timestamp/usage fingerprint. Events
carrying neither are counted rather than dropped, since understating cost is
the worse failure. The correction is reported in the metrics notes.

Add regression tests for the de-duplication, and a dev dependency group so they
run with `uv run python -m pytest tests/`. pytest-asyncio is required by
amplifier-core's pytest plugin, which loads via entry point at startup.

🤖 Generated with [Amplifier](https://github.com/microsoft/amplifier)

Co-Authored-By: Amplifier <240397093+microsoft-amplifier@users.noreply.github.com>
@DavidKoleczek
David Koleczek (DavidKoleczek) merged commit 75ec848 into main Aug 5, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant