Repository navigation
Optional live eval runner. The static scan is unchanged: it still only writes cases, and a scan with no --llm flag still makes no network calls.
- New command
agent-ops-check eval. It validatesagent-evals.yamland calls the agent only with--liveorlive: true. Dry-run is the default. Live mode warns that the agent may take real actions. - Optional Python package
agent-ops-evals(Python 3.11+), built on pydantic-evals 2.59.0 (MIT). Node and Python exchange files only: YAML in,results.jsonout. The Node tool keeps working when Python is not installed. If Python and uv are both missing, the command prints install steps and exits. - Targets in
agent-evals.yaml: an HTTP endpoint, a Python callable, a CLI command, or a Pydantic AI agent. - Packs: LLMail-Inject samples from Microsoft (MIT) in
data/llmail, OWASP-tagged cases generated from the same templates assrc/evals.js, a JSONL file a scan already wrote, and benign emails. - Deterministic yes/no checks: a planted canary leaked, a forbidden tool was called, a secret pattern leaked, a benign case that should be answered was refused (over-block).
- The local-only network promise stays true for the static scan.
evalcontacts only the agent you configure, and nothing else. An opt-in model judge is not in this version. - CI job
python-evalinstalls the package and runs the Python tests against fake HTTP and CLI agents. The existing Node jobs are unchanged. examples/agent-evals.yamldry-runs from a checkout. The generated pack is commented out until you point it at a scan file such asreport/evals/security.jsonl.
Not in this release (later phases): an AI/LLM judge, promptfoo export, and a static-versus-live scorecard.