Proof A: multi-axis forensic-image grader + oracle (no lab) - #37
Conversation
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 279bbdb379
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| account: "accounts", | ||
| host: "hosts", | ||
| file: "file_paths", | ||
| ip: "ip_addresses", |
There was a problem hiding this comment.
When an image oracle plants a web/C2 IOC with ioc.type set to domain or url, the verdict can recover it in indicators.domains or indicators.urls, but this map has no entries for those types. scoreIocs then looks up ind[undefined] and records a MISS even when the IOC is present, so Proof B/C cases with recovered domains or URLs will under-report IOC recall.
Useful? React with 👍 / 👎.
b09ed6c to
06986dc
Compare
…le (no lab) First slice of the forensic-image benchmark: the multi-axis grader and a per-artifact oracle, built and tested as pure code against a REAL captured offline run's actual verdict.json fields — no VM, no forensic toolchain, no image (those are Proof B/C). - scripts/score-image-run.mjs: four axes bound to real fields — MITRE elevation (structured, alias-aware), IOC recovery (recall over only emitted indicator categories; hash/registry marked unscored), chain reconstruction (join to attack_story.attack_chain[] by technique+phase; coverage + ordering), verdict band. Plus an AI-degree detector over documented Tier-1 integration artifacts (embedded keys, ~/.ollama/history, prompts-as-code) and llm_provenance + --require-llm. Fixes inherited bugs (collectCves path; stringify over-count). - fixtures/dfir-image-bench/proofA-winlat/ground-truth.json: hand-authored oracle over the captured win-lateral run (techniques verified vs MITRE G1017; codes stored as version-tagged alias sets). - fixtures/dfir-image-bench/ai-present.verdict.json: crafted fixture proving the AI-degree detector. - scripts/selftest.mjs: 10 assertions wired into CI (full-score positive case, AI-present detection, regression guard, negative control, ordering guard). Full selftest: 176 passed, 0 failed.
279bbdb to
28c5168
Compare
2 participants
What
First slice of the forensic-image benchmark (design:
docs/forensic-image-benchmark-design.mdin the dgx-spark-lab repo). Proof A builds the multi-axis grader + a per-artifact oracle as pure code — tested against a REAL captured offline run's actualverdict.jsonfields, with no VM, no forensic toolchain, and no disk image (those are Proof B/C). Stacked on #36 (the offline scorecard grader it extends).Why this first
An adversarial review of the design found v1 was one all-or-nothing vertical. Proof A retires the genuinely novel part — the machine-readable per-artifact oracle + multi-axis scorers — fastest, with zero lab dependency, and it's CI-testable exactly like #36.
Axes (each bound to fields the producer actually emits — verified)
findings[].mitre_technique/attack_coverage.observed_techniques(not a stringify of narrative, which over-counted).verdict.indicatorsemits (accounts/file_paths/hosts/ip/processes/services);hash/registry_valuereported unscored (no producer channel yet).attack_story.attack_chain[]by technique+phase;coverage+ adjacent-pairordering.~/.ollama/history, prompts-as-code — not writing style) and the honestllm_provenancecontrol +--require-llm.Also fixes two inherited grader bugs (
collectCvesread a non-existentattack_story.targets; technique collection over-counted from narrative text).Test plan
node scripts/selftest.mjs→ 176 passed, 0 failed, including 10 new image-grader assertions: the captured win-lateral run scores full on all four axes (AI-degree = none,deterministic-fallbackprovenance); the craftedai-present.verdict.jsonis flaggedAI-present; a stripped run drops MITRE/IOC/chain (regression guard); an empty verdict scores ~0 (negative control); a reversed chain drops the ordering score.Not in this PR (deferred, honest)
Proof B (install + smoke-test the ARM64 forensic toolchain — 0/18 tools installed today) and Proof C (build one image on an x86 KVM host). This grader is the scoring substrate they feed. Techniques verified vs MITRE ATT&CK G1017; Novel-tier mappings still need a careful pass.