Summary
Findings do not say what kind of file surface they landed on, and triage is much harder without it. This is a request to expose that information, not to filter on it — filtering is contentious for good reasons (see below), while the data itself is useful either way.
The measurement
Scanning 65 real skill/plugin units — 4415 findings, every one triaged by hand — and classifying each finding by the surface its line belongs to:
| rule |
findings |
code |
instructions |
tests |
docs |
comments |
config |
| RP1 |
894 |
262 |
94 |
64 |
381 |
72 |
21 |
| PE3 |
814 |
296 |
22 |
211 |
115 |
156 |
14 |
| AST4 |
417 |
238 |
0 |
178 |
0 |
1 |
0 |
| E1 |
390 |
69 |
7 |
262 |
36 |
7 |
9 |
| RA2 |
235 |
88 |
24 |
24 |
44 |
48 |
7 |
| AS3 |
207 |
7 |
44 |
33 |
48 |
6 |
69 |
| MP2 |
194 |
48 |
20 |
24 |
87 |
15 |
0 |
| EA3 |
175 |
103 |
1 |
8 |
62 |
1 |
0 |
Totals across all 53 rules: code 1442 · tests 981 · docs 980 · comments 412 · instructions (SKILL.md) 325 · config 275.
instructions is kept separate from docs on purpose: SKILL.md is not prose about a command, it is what the agent executes — which is exactly why _is_documentation_context already excludes it. Counting it as documentation overstates the "non-executable" share, and I made that mistake before correcting it.
Why exposing, and not filtering
Two of these columns look like obvious suppression candidates. They are not:
tests (981 findings). A fixture containing an attack string is expected — but a directory named tests/ is also exactly where a payload would hide. Exempting it trades a real detection for a cosmetic gain.config — and the reason the aggregate is misleading. SC1 has 100% of its findings outside executable code, entirely in config files. For a dependency-pinning rule that is the correct place. "Not code" does not mean "noise", and the ratio only means something per rule.
Both arguments point the same way: the classification is genuinely useful for whoever reads the report, and genuinely dangerous as an automatic filter. So ship the data and let the consumer decide.
What this would look like
A normalised field on each finding — surface: code | instructions | docs | tests | config — derived from the path (and, for the comment case, from the matched line). The runner already computes file_type and _infer_file_type, so most of the input exists.
Concretely it would let a consumer:
- sort a 400-finding report by surface and read the 238
code ones first;
- write a suppression policy locally, taking on the risk explicitly, instead of the scanner taking it on their behalf;
- diff two scans by surface, which is how one notices that a rule started matching documentation after a version bump — how the
PE3 change in 2.4/2.5 became visible here at all.
I have not opened a PR because this touches the report schema, which is a public contract and yours to design. If the shape is welcome I am happy to implement it; if the answer is "the consumer can derive this from location.file", that is a fair answer too — it is what I ended up doing, and the reason I can post the table above.
Context
From the same corpus that produced #302, #319, #321, #322 and #323. Zero true positives in 4415 findings, which is a statement about this corpus (first-party and well-known marketplaces), not about the scanner.
Summary
Findings do not say what kind of file surface they landed on, and triage is much harder without it. This is a request to expose that information, not to filter on it — filtering is contentious for good reasons (see below), while the data itself is useful either way.
The measurement
Scanning 65 real skill/plugin units — 4415 findings, every one triaged by hand — and classifying each finding by the surface its line belongs to:
Totals across all 53 rules: code 1442 · tests 981 · docs 980 · comments 412 · instructions (
SKILL.md) 325 · config 275.instructionsis kept separate fromdocson purpose:SKILL.mdis not prose about a command, it is what the agent executes — which is exactly why_is_documentation_contextalready excludes it. Counting it as documentation overstates the "non-executable" share, and I made that mistake before correcting it.Why exposing, and not filtering
Two of these columns look like obvious suppression candidates. They are not:
tests(981 findings). A fixture containing an attack string is expected — but a directory namedtests/is also exactly where a payload would hide. Exempting it trades a real detection for a cosmetic gain.config— and the reason the aggregate is misleading.SC1has 100% of its findings outside executable code, entirely in config files. For a dependency-pinning rule that is the correct place. "Not code" does not mean "noise", and the ratio only means something per rule.Both arguments point the same way: the classification is genuinely useful for whoever reads the report, and genuinely dangerous as an automatic filter. So ship the data and let the consumer decide.
What this would look like
A normalised field on each finding —
surface: code | instructions | docs | tests | config— derived from the path (and, for the comment case, from the matched line). The runner already computesfile_typeand_infer_file_type, so most of the input exists.Concretely it would let a consumer:
codeones first;PE3change in 2.4/2.5 became visible here at all.I have not opened a PR because this touches the report schema, which is a public contract and yours to design. If the shape is welcome I am happy to implement it; if the answer is "the consumer can derive this from
location.file", that is a fair answer too — it is what I ended up doing, and the reason I can post the table above.Context
From the same corpus that produced #302, #319, #321, #322 and #323. Zero true positives in 4415 findings, which is a statement about this corpus (first-party and well-known marketplaces), not about the scanner.