test(detection): add benign corpus coverage - #145
Conversation
|
All contributors have signed the CLA ✍️ ✅ |
|
Thank you for this. Benign corpus cases are worth more to this project than new rules, and these three are well chosen: authenticated registry egress, a Terraform apply with a named cloud profile, and a short dependency-audit network burst are exactly the shapes that make sequence rules cry wolf on ordinary work. I have approved the CI run. I want to ask about the CLA rather than nudge you about it again. You closed #140 rather than sign, and that one was finished. All eight checks passed and it was better than the issue asked for. So I am reading that as a deliberate decision rather than an oversight, and nudging you a second time would be rude. What is the blocker? If it is an employer restriction, or a policy against copyright assignment, or an objection to CLAs generally in favour of a DCO, I would rather know than guess. Those are all reasonable positions and several are ones I would hold myself in your place. I am not promising to change it, because the CLA exists for a real reason: this project has an open core and a separate commercial layer, and the CLA is what keeps relicensing possible. That is a genuine constraint rather than paperwork for its own sake. But it has now cost one finished contribution and is about to cost a second, and that is a fact about the trade rather than about you. If a DCO would work where a CLA does not, that is worth knowing, and it is a decision I should make with evidence rather than in the abstract. If you would rather not say, that is completely fine and no explanation is owed. In that case tell me and I will close this with thanks rather than leave it sitting blocked, and I will write the three corpus cases myself and credit you in the commit for identifying the shapes. Your work would not be wasted either way. Sorry about #140. It was good, and losing it was avoidable. |
|
I have read the CLA Document and I hereby sign the CLA |
|
Thank you for writing this out so thoughtfully, and I’m sorry I left you guessing. It was not a deliberate objection to this project or to the CLA. I hesitated around the process and then handled it poorly by closing #140 instead of saying that clearly. Reading your note made me realize that this created unnecessary uncertainty for you and also threw away work that you had already taken the time to review carefully. I’ve signed the CLA for this PR now. I would also like to reopen #140 and sign there as well, if you are still open to reviewing it. That PR mattered to me too, and I’d rather not leave a finished contribution discarded because I handled the process badly. I appreciate the direct explanation of why the CLA matters for this project, and also the generous way you framed the alternatives. I should have communicated earlier. Thank you for giving me the chance to fix it. |
There was a problem hiding this comment.
🟡 Changes recommended
One new corpus event uses a MITRE tactic/technique combination that is inconsistent with the repository’s MITRE mapping expectations, reducing fixture fidelity.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
Expands the detection benchmark’s benign corpus to better represent “normal but noisy” developer workflows (registry publish, Terraform apply with named profile, dependency audit burst), and updates public-facing documentation to reflect the new corpus size.
Changes:
- Adds 3 new benign detection-corpus sessions and registers them in
corpus.yaml. - Updates benchmark session counts across SECURITY/README/ROADMAP/whitepaper to 57 total sessions (31 benign).
- Documents the rationale for why each new benign session might look suspicious but should not fire.
File summaries
| File | Description |
|---|---|
| SECURITY.md | Updates the benchmark size reference used as part of the security controls narrative. |
| ROADMAP.md | Updates benchmark corpus size and benign/attack split. |
| README.md | Updates the sample benchmark output to reflect current corpus size. |
| docs/whitepaper.md | Updates benchmark corpus size and benign/attack split in the credibility section. |
| apps/orchestrator/agentmetry/core/audit/detection/corpus/corpus.yaml | Registers the three new benign sessions with notes and expected empty detections. |
| apps/orchestrator/agentmetry/core/audit/detection/corpus/benign_container_registry_publish.jsonl | Adds a benign session for docker login/build/push with registry egress traits. |
| apps/orchestrator/agentmetry/core/audit/detection/corpus/benign_terraform_apply_with_profile.jsonl | Adds a benign Terraform apply session using a named AWS profile and a cloud API call. |
| apps/orchestrator/agentmetry/core/audit/detection/corpus/benign_dependency_audit_network_burst.jsonl | Adds a benign dependency-audit session with short network burst and local summarization. |
Review details
- Files reviewed: 8/8 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| @@ -0,0 +1,4 @@ | |||
| {"action":{"outcome":"success","reason":"decision:allow;hook:PreToolUse","type":"tool_called"},"actor":{"id":"operator","role":"operator","type":"agent"},"agent":{"name":"claude","skill_id":""},"correlation_id":"b-terraform-apply","event_id":"8b7001f1-6c5b-51ab-92e7-8a72c8f6d001","fleet_id":"example-fleet","host_id":"BUILD-01","initiator":{"actor_type":"agent","operator_id":"operator","trigger":"manual"},"model":{"id":"claude","provider":"claude"},"schema_version":"1.1.0","session_id":"b-terraform-apply","source":{"adapter":"claude_hook","app":"claude","tier":"external"},"source_topic":"external/claude/tool_called","timestamp_utc":"2026-08-12T11:00:00+00:00","tool":{"arguments":{"command":"terraform init -backend-config=env/staging.backend"},"command":"terraform init -backend-config=env/staging.backend","input_hash":"4c6e8f1028dd96c5d2e9b8a6a722f78c87b80ac3e4d2cc9d7f5d3a44a9148d04","input_redaction":"hash+command","mitre":{"tactic":"Execution","tactic_id":"TA0002","technique":"Unix Shell","technique_id":"T1059.004"},"name":"Bash","parameters_redacted":false,"qualified":"Bash","server":"claude"}} | |||
| {"action":{"outcome":"success","reason":"decision:allow;hook:PreToolUse","type":"tool_called"},"actor":{"id":"operator","role":"operator","type":"agent"},"agent":{"name":"claude","skill_id":""},"correlation_id":"b-terraform-apply","event_id":"8b7001f1-6c5b-51ab-92e7-8a72c8f6d002","fleet_id":"example-fleet","host_id":"BUILD-01","initiator":{"actor_type":"agent","operator_id":"operator","trigger":"manual"},"model":{"id":"claude","provider":"claude"},"schema_version":"1.1.0","session_id":"b-terraform-apply","source":{"adapter":"claude_hook","app":"claude","tier":"external"},"source_topic":"external/claude/tool_called","timestamp_utc":"2026-08-12T11:02:00+00:00","tool":{"arguments":{"command":"terraform plan -var-file env/staging.tfvars -out tfplan"},"command":"terraform plan -var-file env/staging.tfvars -out tfplan","input_hash":"5d7f901128dd96c5d2e9b8a6a722f78c87b80ac3e4d2cc9d7f5d3a44a9148d05","input_redaction":"hash+command","mitre":{"tactic":"Execution","tactic_id":"TA0002","technique":"Unix Shell","technique_id":"T1059.004"},"name":"Bash","parameters_redacted":false,"qualified":"Bash","server":"claude"}} | |||
| {"action":{"outcome":"success","reason":"decision:allow;hook:PreToolUse","type":"tool_called"},"actor":{"id":"operator","role":"operator","type":"agent"},"agent":{"name":"claude","skill_id":""},"correlation_id":"b-terraform-apply","event_id":"8b7001f1-6c5b-51ab-92e7-8a72c8f6d003","fleet_id":"example-fleet","host_id":"BUILD-01","initiator":{"actor_type":"agent","operator_id":"operator","trigger":"manual"},"model":{"id":"claude","provider":"claude"},"schema_version":"1.1.0","session_id":"b-terraform-apply","source":{"adapter":"claude_hook","app":"claude","tier":"external"},"source_topic":"external/claude/tool_called","timestamp_utc":"2026-08-12T11:04:00+00:00","tool":{"arguments":{"command":"AWS_PROFILE=staging terraform apply -auto-approve tfplan"},"command":"AWS_PROFILE=staging terraform apply -auto-approve tfplan","input_hash":"6e80912228dd96c5d2e9b8a6a722f78c87b80ac3e4d2cc9d7f5d3a44a9148d06","input_redaction":"hash+command","mitre":{"tactic":"Impact","tactic_id":"TA0040","technique":"Data Manipulation","technique_id":"T1565"},"name":"Bash","parameters_redacted":false,"qualified":"Bash","server":"claude"}} | |||
|
Small update on the related PR: I reopened #140, rebased it onto current master after #139, and marked it ready for review. The PR is mergeable now, and the local CLI tests plus Ruff pass on the rebased head. I also tried to post the CLA signing phrase on #140, but GitHub rejects new comments there because the conversation is locked. If you can unlock the thread or retrigger the CLA check another way, I can complete that part there too. |
`test_funding_manifest.py` pinned the benchmark case count in `funding.json` to `len(load_corpus())`. The next contributor pull request to add corpus cases, blitzcrieg1#145, went red on both platforms with `assert 54 == 57`, against a file the contributor had never touched and had no business editing. The guard worked. It was in the wrong place. A public claim of mine is not a contributor's problem to fix, and a red check on their branch for it is a tax they cannot pay. So the manifest no longer quotes counts at all, and this no longer checks them. What stays pinned is what moves on a release rather than on a contribution: the version and the licence, both of which change in a commit I make anyway. The counts live in the README, where `test_readme_claims.py` guards them and the change and the claim arrive in the same pull request. This is the same rule CONTRIBUTORS.md landed on yesterday, applied consistently this time: stamp a number with a date when it moves on its own, pin it only when the person changing it is the person who can fix it. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
What
Refs #25. Adds three benign detection-corpus sessions for registry publishing, Terraform apply, and dependency audit workflows.
Fix
The new cases cover normal work that looks noisy to sequence rules: authenticated registry egress, infrastructure changes with a named cloud profile, and short dependency-audit network activity.
Test
Adds corpus entries that keep the benchmark at 0 missed and 0 false positives.