Session-level evals
- Multi-turn eval cases:
inputmay be a list of turns; they run as one session, so later turns see earlier history and recovery across a conversation is evaluable. - New
handoffgrader: passes only if the session actually escalated to a human, so "hands off on hardship" is testable end-to-end (5 graders total: exact, contains, tool_called, handoff, judge). - Judge grades in full context: the agent's system prompt and the whole conversation, with one line of reasoning before the verdict, kept in the report as
judge_output. - Per-cohort
strictness:strict(default, any doubt fails; right for compliance cohorts) orlenient(for tone/style cohorts). - Judge independence: the runner warns when the judge model is the agent's own model; docs recommend a different model family.
Docs and playbooks
- Playbooks updated to gpt-5.6: agents on
gpt-5.6-sol, judges ongpt-5.6-terra; escalation cases moved to session-levelhandoffcohorts.
All changes are backward compatible with existing cohort YAML.