test(dfir): offline scorecard regression guard (fixture-driven) - #19
Conversation
Companion to caseforge-core's offline gpt-oss:20b DFIR test. verdict-opencode is the runtime the investigate pipeline drives; it does not produce verdict.json itself, so this guard is fixture-driven: it scores a real, checked-in offline sealed run against the ground-truth oracle with no LLM and no network. - packages/opencode/test/dfir/grader.ts: typed technique/CVE grader + provenance. - packages/opencode/test/dfir/scorecard.test.ts: bun:test asserting full HITs, the expected verdict band, and that the seal is a deterministic fallback (the gpt-oss:20b agent was forced but fell back on invalid tool calls, so the LLM authored none of it) — plus a regression guard. - fixtures/: ground-truth oracle + the real win-lateral-movement sealed run.
|
Thanks for your contribution! This PR doesn't have a linked issue. All PRs must reference an existing issue. Please:
See CONTRIBUTING.md for details. |
|
This PR doesn't fully meet our contributing guidelines and PR template. What needs to be fixed:
Please edit this PR description to address the above within 2 hours, or it will be automatically closed. If you believe this was flagged incorrectly, please let a maintainer know. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 136158a5ef
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| if (tgt?.technique_id && /finding|elevat|confirm|observ/i.test(String(tgt.status ?? ""))) { | ||
| set.add(String(tgt.technique_id).toUpperCase()) |
There was a problem hiding this comment.
When a report has an attack_coverage.targets entry whose status is covered_no_finding for an expected technique, but the technique is absent from observed_techniques and findings, this regex still adds it because the status contains finding. scoreRun then upgrades a non-elevated coverage target to HIT, inflating the offline scorecard and masking exactly the kind of missing-elevation regression this guard is meant to catch.
Useful? React with 👍 / 👎.
| export function collectCves(verdict: any): Set<string> { | ||
| const set = new Set<string>() | ||
| const add = (s: string) => s.match(CVE)?.forEach((c) => set.add(c.toUpperCase())) | ||
| for (const tgt of verdict.attack_story?.targets ?? []) (tgt.cves ?? []).forEach((c: string) => add(String(c))) |
There was a problem hiding this comment.
For verdicts that carry CVEs in attack_story.attack_chain without duplicating them in findings or top-level cves, this loop never sees them because the checked-in verdict shape uses attack_chain, not attack_story.targets. In that scenario scoreRun marks an expected CVE as MISS even though the attack story contains it, so the scorecard can undercount CVE coverage depending on where the seal emits the CVE.
Useful? React with 👍 / 👎.
2 participants
What
Companion to caseforge-core #36 (the live offline gpt-oss:20b DFIR test + grader). verdict-opencode is the runtime the caseforge investigate pipeline drives — it does not produce
verdict.jsonitself — so this is a fixture-driven regression guard: it scores a real, checked-in offline sealed run against the ground-truth oracle with no LLM and no network.Why it matters
The offline DFIR run forces
gpt-oss:20bas the agent (CASEFORGE_FORCE_AGENT=1), but the model fell back to the deterministic engine after repeated invalid tool calls. This guard pins that honest fact at the runtime layer: it asserts the seal is a deterministic fallback, so a future change that silently claimed LLM authorship would fail the test.Changes (all under
packages/opencode/test/dfir/)grader.ts— typed technique/CVE grader +detectProvenance(llm-agent vs deterministic-fallback).scorecard.test.ts—bun:testasserting full technique elevation (3/3), the expected verdict band,deterministic-fallbackprovenance, LLM-authored recognition, and a detection-regression guard.fixtures/— ground-truth oracle + the realwin-lateral-movementsealed offline run.Test plan
cd packages/opencode && bun test test/dfir→ 5 pass, 0 fail. Lint (oxlint) and typecheck (tsgo) pass. No LLM, no network — safe in CI.