-
Notifications
You must be signed in to change notification settings - Fork 1
Description
Problem
The scan pipeline is strictly additive — each layer can only add findings, never suppress or demote findings from earlier layers. This means the LLM analyzer's contextual understanding is wasted when it correctly identifies rule engine findings as false positives.
Example: securityreview skill on ClawHub
malwar crawl scan securityreview
Result: MALICIOUS (100/100) — but the findings are FPs:
| Severity | Rule | What it found |
|---|---|---|
| HIGH | MALWAR-CRED-001 | AKIAIOSFODNN7EXAMPLE (AWS example key) |
| HIGH | MALWAR-CRED-001 | wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY (AWS example key) |
| LOW | llm-deception | "Prior Detector Flags Are False Positives from Example Code" |
The LLM correctly identifies the credentials as AWS's well-known placeholder keys used in documentation, but it has no mechanism to demote or suppress the HIGH findings from the rule engine. The final score remains 100/100 despite the LLM's correct assessment.
Proposed Solution
Add a feedback path from the LLM analyzer back to the risk scoring layer:
- When the LLM identifies prior findings as false positives (example code, documentation patterns, test fixtures), it should emit a suppression annotation referencing the original finding IDs
- The risk scorer should process suppression annotations by reducing confidence or removing suppressed findings from the score calculation
- Suppressed findings should still appear in the output (for transparency) but marked as
suppressedwith the LLM's reasoning
Impact
Without this, security-focused skills that reference example credentials, document attack patterns, or include test vectors will consistently score as MALICIOUS — reducing trust in scan results and causing alert fatigue.