feat: surface the non-exhaustiveness disclosure in the human PR summary#20
Merged
Conversation
The native detector lane scanned bundled third-party code (pip/_vendor, vendor/, third_party/, site-packages). On pip, 280 of 362 native findings (77%) came from _vendor/ — code the project ships but does not own and cannot fix. Add _vendor, vendor, vendored, third_party, third-party, site-packages, bower_components to DEFAULT_GENERATED_PATH_SEGMENTS. After the fix pip reports 74 findings, 0 from _vendor, while its own code is still scanned. This is the native twin of the eslint built-JS fix: both lanes were scanning code that is not the project's own source. Locked by test_vendored_third_party_code_is_not_scanned; sample-report mirrors and ACI_EVIDENCE.md (§2i) updated. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…policy §2h (eslint) and §2i (native) were the same defect — scanning code that is not the project's own source — fixed twice because each lane kept its own skip set and they had drifted apart. The borrowed-analyzer skip sets were each missing ten segments the native policy already had, including every vendored one (_vendor, vendor, vendored, third_party, site-packages, bower_components). The report already dropped these (external findings are bounded to the native-discovered file set), but the borrowed tools still processed the code: on a pip-scale _vendor/ tree, handing hundreds of bundled files to mypy can exhaust the per-analyzer timeout and starve the project's own code of findings. Derive _PYTHON_ANALYZER_SKIP_SEGMENTS / _SEMGREP_SKIP_SEGMENTS / _ESLINT_SKIP_SEGMENTS from DEFAULT_GENERATED_PATH_SEGMENTS plus lane- specific extras, so they cannot drift again. Add a drift-guard test and a vendored file-discovery test; ACI_EVIDENCE.md §2j records the root cause. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Reviewing a branch that deletes files reports the deleted path in git's changed set while it is gone from the working tree. Verified by a real git rename+delete probe that the scan handles this gracefully (the deleted file is simply absent from the rglob'd target set); this test locks that guarantee so a future refactor cannot reintroduce a crash or a phantom finding for a vanished file. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The operations baseline is the backbone of the "only new issues" review, and the source of three earlier defects (all from encoding a finding's identity by line number). Verified by a real scan->baseline->edit->rescan probe that the workflow is sound: - existing findings keep their identity after an unrelated line shift (fingerprint match short-circuits the stale line in the baseline entry), - a genuinely new finding is flagged new, - a fixed finding is reported resolved. This end-to-end test ties fingerprint stability to operations matching in one guarantee, where the prior tests covered each half separately. The waiver path (active-waiver -> accepted/accepted-residual) was probed too and is already covered by existing tests. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…scan crash ACI ingests JSON/SARIF from 13 external tools. The parse sites caught only json.JSONDecodeError, so output that was well-formed but shape-shifted (a renamed key, a null where a table was, a list that became an object -- the ordinary result of a tool version bump) raised KeyError/TypeError/ AttributeError instead. That escaped the parse-failure path and aborted the whole scan, taking the native findings and every other analyzer down with one borrowed tool's drift. Verified: raw ruff/eslint/trivy/osv output in a drifted shape, plus codeql SARIF with a wrong-typed `runs` and a gitleaks report with a wrong-typed field, all crashed before the fix. Broaden the three parse sites (stdout JSON, gitleaks report, codeql SARIF) to a shared _ANALYZER_PARSE_ERRORS set so any structural mismatch becomes a recorded parse-failure for that one analyzer while the scan continues. One definition, used at all three sites, so the lanes cannot drift apart. Regression tests cover the drift cases and confirm valid output still parses. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
ACI's purpose includes making users aware that even its declared checks are not 100% -- a clean result does not prove the absence of issues. That disclosure already rides in the JSON report, but the GitHub PR summary, the surface where a reviewer actually concludes "ACI passed, the code is clean," omitted it. A zero-finding pass is exactly where that false confidence is highest, yet the summary said only "Gate: pass / Findings: 0." Add the disclosure as a footer to build_github_summary_markdown, sourced from the report's own detection_disclosure key (single source of truth), so the honest bound travels to the point of use. Reports without the key still render. This serves the "make the bound explicit, at the point of use" half of ACI's purpose, which until now lived only in docs and the machine report. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
概要
ACI の目的の一つは「ACI の宣言済みチェックでも検知は100%ではない —— クリーンな結果が問題の不在を証明しない」と利用者に認知させることです。この開示は JSON レポートには既に載っていましたが、レビュアーが実際に「ACI は pass、コードはクリーン」と判断する面 —— GitHub の PR 要約 —— からは欠落していました。
0件・pass のときこそ誤った安心が最大になるのに、要約は「Gate: pass / Findings: 0」としか伝えていませんでした。これは目的に対する明確なギャップです（検知能力ではなく、誠実な境界の可視化の欠落）。
是正
build_github_summary_markdownに開示を脚注として追加しました。出所はレポート自身のdetection_disclosureキー（単一の出所）で、誠実な境界が使用時点まで届くようにしています。キーが無いレポートも引き続き描画されます。これは ACI の目的のうち「境界を明文化し、使用時点で示す」半分を満たすもので、これまで文書と機械可読レポートにしか存在しませんでした。
検証
🤖 Generated with Claude Code