Accuracy

Reproducible accuracy measurement

How we measure detection accuracy — and why the scores live in the repository, not on this page.

---

The four metrics

Metric	What it measures
Recall	share of ground-truth findings the agent surfaced
False positive rate	claims unsupported by the bundled evidence
Hallucination count	facts not present in the source artifacts at all
Evidence integrity	SHA-256 of every input file, before and after the run

Every finding the agent emits carries the audit_ids of the MCP calls that produced the supporting evidence, and any finding can be traced back to those logged calls with python3 -m dfir_audit trace <audit.jsonl> <finding_id>. A missing finding lowers recall; a claim that cannot be traced to a logged call is what the hallucination count measures.

---

Where the numbers live

Benchmark scores are deliberately kept in one place — the repository's docs/benchmarks/, regenerated from a live run rather than transcribed by hand. This page documents how accuracy is measured and what holds regardless of the score. For the latest measured recall across models and cases, read the benchmark files themselves:

• docs/benchmarks/SUMMARY.md — combined recall per model
• docs/benchmarks/MODEL-COMPARISON.md — per-case, per-model breakdown
• docs/benchmarks/HISTORY.md — run-over-run history

Recall varies by case difficulty and by model — that variation is the honest signal. Pinning a single figure on a wiki page would only invite it to drift out of sync with the harness. Reproduce any published number locally:

git clone https://github.com/Juwon1405/agentic-dfir.git
cd agentic-dfir
export PYTHONPATH="$PWD/dfir_audit/src:$PWD/dfir_mcp/src:$PWD/dfir_agent/src:$PWD/dfir_corr/src"
python3 -m scripts.eval.demo       # deterministic baseline, no API key
python3 -m scripts.eval.self       # live measurement, needs ANTHROPIC_API_KEY
python3 -m scripts.eval.external   # public datasets (downloads them first)

---

What does not change — the invariants

Whichever case or model you run, these hold by construction, not by tuning:

• Every finding is traceable. Every finding carries the audit_ids of the MCP calls that produced it, and dfir_audit trace follows a finding back to the exact logged calls. An invented fact therefore cannot hide behind the report — it shows up as a claim with no matching audit entry, which is what the hallucination count measures. A low recall means missed coverage; the two metrics are independent.
• Evidence integrity is sealed. SHA-256 of every input file is recorded before and after each run, and the audit trail is hash-linked into an unbroken chain.
• The read-only boundary holds. The MCP surface exposes only typed, read-only forensic functions — no shell, no eval, no write path. Asserted on every commit:

Test	Result
Surface is exactly the documented read-only function set	PASS
execute_shell raises ToolNotFound	PASS
eval, exec, subprocess_run all raise ToolNotFound	PASS
_safe_resolve rejects .. traversal	PASS
_safe_resolve rejects absolute paths outside DFIR_EVIDENCE_ROOT	PASS
_safe_resolve rejects null-byte truncation attacks	PASS

A random fuzz against destructive function names is blocked on every attempt — the architecture-first guarantee, not a prompt instruction. See Architecture deep dive.

---

Needle-in-a-haystack, not toy data

A fair reviewer asks: "recall on a 30-line file is meaningless — every line is an IOC." Correct. The canonical bundled evidence root (examples/case-studies/self-evaluation/case-01/evidence_root/) is hand-curated at production volume — a security EventLog of roughly eleven thousand lines, supply-chain artifacts, RDP brute-force, USB setupapi — and the two IOC-only logs are enriched with deterministic benign noise (deterministic, committed as-is) to a heavy signal-to-noise ratio. The measurement is not a small-input over-fit; the agent finds the needle in production-scale hay.

---

What this report is not claiming

• Not that the agent matches a senior human analyst on open-ended novel cases. It matches on cases with mechanically verifiable ground truth.
• Not zero false negatives in adversarial settings — only against the documented corpus.
• Not production-readiness. This release demonstrates that the architecture is correct and the loop is sound; hardening is the Phase 2–3 roadmap.

---

External benchmarking — the paradigm gap, honestly

The synthetic measurement is necessary but not sufficient. The honest reviewer question — "what does it score on a dataset you didn't author?" — is answered by integrating external corpora (NIST CFReDS Hacking Case, Ali Hadi Challenge 1, Digital Corpora M57). Those scores live in docs/benchmarks/ alongside the synthetic ones.

The point worth making here is why external recall sits below synthetic recall — and it is not a regression:

• Synthetic accuracy measures correctness of the detection logic against IOCs the system claims to detect.
• External accuracy measures expansion potential against a content-centric paradigm dfir-mcp is still building out.

External benchmarking is what converted "we should add registry parsing someday" into "registry parsing unblocks several measured findings — ship it next." That is the real value of third-party data: it reorders the Phase 2 backlog by evidence, not by guess.

---

See also

• examples/case-studies/external-evaluation/case-01/README.md — external case study
• docs/accuracy-report.md — full accuracy report with current measured scores