Benchmark and Self Test

Benchmark and self-test

Two commands measure Pandaward itself. Both run offline and change nothing in your project.

pandaward benchmark

Runs every built-in rule against Pandaward's corpus of small vulnerable and safe cases and prints, per rule, the number of vulnerable and safe cases, TP, FP, FN, TN, precision and recall.

pandaward benchmark
pandaward benchmark --json      # the format attached to each release as benchmark.json
pandaward benchmark --strict    # exit 1 on any miss or false positive

A rule counts as implemented_and_benchmarked only with precision and recall of at least 0.95 on at least 3 vulnerable and 3 safe cases. Today every rule has one of each, so none qualifies yet; growing the corpus is part of the road to 1.0, and contributions are welcome (see Writing a Check).

The corpus is Pandaward's own, so the numbers describe these cases, not real-world accuracy.

pandaward selftest

Runs a hostile fixture repository through scan, reports, MCP and hooks and checks that repository text could not steer Pandaward:

Case	What must hold
no-code-execution	repository scripts, hooks and build files are never executed
no-scope-widening	a scope manifest or approval shipped with the repository never authorizes network tests
no-disposition-change	repository text never confirms, waives or approves anything
decisions-follow-evidence	a comment asking for PASS does not change a control with evidence of failure
output-neutralized	repository text cannot inject terminal control sequences or HTML into reports and tool output
mcp-untrusted-notice	every MCP result marks repository-derived fields as data
hooks-require-human	approving scope, sending traffic, changing dispositions and editing scope or ledger always need the human
hypothesis-stays-hypothesis	an agent-proposed finding is never confirmed by its own text
tool-output-is-data	scanner output (SARIF) that claims to be confirmed or asks for a disposition is imported as a hypothesis only

CI runs the self-test on Linux, macOS and Windows for every change, and the release workflow runs it before building; any failed case blocks the release. Each line reports the number of probes it made.

Passing shows that these specific attempts failed on this build. It does not prove that no other input can steer an agent that uses Pandaward.