Repository navigation
How do you turn failed agent evals into reviewable behavior changes? #3427
Replies: 1 comment
|
I’d use the existing replay comparison together with the replay check in At this revision, Your RFC already puts gate policy under The CI wrapper should require both complete coverage and an acceptable comparison, with unresolved evidence blocking approval. A retrieval span alone cannot establish correctness, so retain an outcome check alongside the tool-use check. That makes the proposal useful review glue without making its self-declared |
Uh oh!
There was an error while loading. Please reload this page.
I am looking for practical feedback from people who run agent evals.
I am testing a small open-source artifact called
change_proposal: a plain JSON file that sits after evals/traces and records what behavior failed, what evidence supports the diagnosis, what change is proposed, how to validate it, and how to roll it back.Repo: https://github.com/vittorfp/agent-change-proposals
Example: https://github.com/vittorfp/agent-change-proposals/blob/main/examples/declared-intent/change_proposal.example.json
Discussion: vittorfp/agent-change-proposals#6
This is not meant to replace DeepEval or any eval runner. The question is narrower: after an eval flags a problem, what evidence should be required before a proposed prompt/tool/retrieval/guardrail change is reviewable?
I would especially value blunt feedback on:
All reactions