Skip to content

v0.6.0 - Toolscore diagnostics and native-agent case studies

Latest

Choose a tag to compare

@guybass guybass released this 03 Oct 12:32

Agent Eval Flow 0.6.0 adds deterministic tool-call evaluation and offline case studies, and fixes two failures encountered in native-agent captures.

Changes

  • Optional Toolscore 1.9.0 integration with eight diagnostics, including identical-call rate, source-linked receipts, and portable tools reports alongside the general report. Correct empty traces receive a perfect native score; incomplete traces stay unknown.
  • Claude stream projection accepts system and result records without discarding the retained stream. Unsupported assistant/user message content still fails explicitly. (#5)
  • Completed grading survives a backwards host-clock adjustment. (#4)
  • Offline GPT Researcher, DeerFlow, and MCP memory examples import retained evidence and keep outcomes separate from tool diagnostics. MCP importers preserve tool errors and use the bundled scenario file. (#1, #2, #3)
  • DeerFlow summaries retain observed request counts when the optional Toolscore package is absent, with the missing scorer reported explicitly.
  • Security-triage research documentation records uncertainty loss, development regressions, failed attempts, and evidence limitations. (#8)
  • Publishing verifies all twelve offline and Toolscore CI jobs across Windows/Linux and Python 3.11, 3.12, and 3.13. Regression tests reject missing, skipped, duplicated, or failed job coverage.

Upgrade

python -m pip install --upgrade "agent-eval-flow[toolscore]==0.6.0"

Toolscore is optional. The base installation and rendering saved reports do not require it. Existing 1.8.1 receipts keep their original scores. For new comparisons, regrade both candidates with the same evaluator: one-to-one argument pairing and correct no-tool scoring can change existing scores.

Validation and scope

The release passes the offline and Toolscore CI matrices, package integrity and metadata checks, and offline report examples. All 85 frozen acceptance files retain their original hashes. The included studies are documented development evidence, not a new untouched holdout or a general reliability estimate; this release performs no new live-agent evaluation.

The contract-v2 and Claude subscription-auth proposal (#7) is deferred pending compatibility, report-count, and isolation validation. Those experimental APIs are not included in 0.6.0; the original experimental branch and frozen study remain unchanged.

Merged contributions: #2, #3, #4, #5, #6, and #8. The existing Toolscore integration and GPT Researcher changes are included in this first PyPI release since 0.5.1.