Skip to content

2.3.1

Choose a tag to compare

@not-so-fat not-so-fat released this 29 Oct 23:18
· 41 commits to main since this release

Feature: evaluation scenarios (NOT-535)

  • New dashboard-authenticated REST API for evaluation scenarios, manually curated feedback-derived cases for one target playbook: POST/GET /api/evaluation-scenarios, GET/PUT /api/evaluation-scenarios/:id. A scenario can only become ready with task input, a historical failed output, at least one existing feedback signal and one deterministic expectation. Reference in docs/EVALUATION_SCENARIOS_API.md.
  • The Feedback page can now curate evaluation scenarios from selected signals with a scenario editor. Expectation types match the agent-deck experiment evaluate evaluator.