ASV Eval

ASV Eval measures which steps in an agent trace helped.

Input: agent trajectory, candidate outcomes, and evaluator scores. Output: per-step entropy movement, gold-margin gain, Bayesian surprise, and a local report bundle.

Install

pip install -e ".[dev]"

Quickstart

asv evaluate \
  --input examples/provided_beliefs/trajectory.jsonl \
  --belief-fixture examples/provided_beliefs/beliefs.jsonl \
  --output-dir /tmp/asv-report

The command writes summary.json, steps.jsonl, states.jsonl, CSV tables, and report.md.

Commands

asv evaluate --help
asv adapt-open-qa --help
asv adapt-react --help
asv audit-permutations --help

Live scoring uses DeepSeek chat log probabilities:

export DEEPSEEK_API_KEY=<your-key>
asv evaluate \
  --input trajectories.jsonl \
  --evaluator deepseek-chat-logprob \
  --rationale-mode label-free \
  --fallback-policy floor \
  --cache .asv-cache.jsonl \
  --run-ledger .asv-run-ledger.jsonl \
  --output-dir report

No network access is needed for the provided-belief example or the test suite.

Input Format

ASV Eval uses standard ASV JSONL with one trajectory per line. See docs/schema.md.

Evaluators

Use --belief-fixture for deterministic offline runs. Use --evaluator deepseek-chat-logprob when you want ASV Eval to score missing beliefs with DeepSeek log probabilities. See docs/evaluator_protocol.md.

Report Bundle

The report bundle is documented in docs/report_bundle.md.