Repository navigation
How are you checking agent evals against known-bad outcomes? #3651
hector2394
started this conversation in
General
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I'm comparing existing eval tools before deciding whether another tool is needed. Local keyless fixtures through muteval and acyclic-eval showed that both can evaluate authored good/bad cases once the state and expectations are supplied.
On your last scorer change, how did you obtain trustworthy negative cases or service-state evidence? What, if anything, still needed awkward manual work after using existing tools?
Public or sanitized examples would be helpful.
All reactions