feat(agent-ux): confidence scores per finding + schema_version in JSON output (closes #5)#148
Merged
Merged
Conversation
…N output (closes #5) Two agent-UX features from issue #5, implemented as a non-breaking enhancement so existing consumers keep working: 1. Confidence scores (sub-feature #1) - New scripts/confidence.py: category-driven scoring with modifier adjustments. Dead-code categories: unreachable=0.95, unused_vars=0.70, zombie_css=0.50, unused_exports=0.45, registry_dead=0.75. Secrets categories: pattern_match=0.90, env_exposed=0.85, entropy=0.65. Modifiers: test files -0.15..-0.30, config files -0.10, library API -0.20, downgraded -0.05..-0.10, after=return +0.03, after=throw -0.03. - deadcode_engine.detect_dead_code() enriches every finding with a 'confidence' field via enrich_findings('dead_code', payload). - secrets_engine.detect_secrets() does the same for secrets findings. - Defensive import: if confidence module is unavailable, engines still return valid payloads (fail-safe). - confidence_distribution() helper buckets findings into high/medium/ low/unscored for quick reliability summaries. 2. Schema versioning (sub-feature #5) - SCHEMA_VERSION = '8.2' in scripts/confidence.py (single source of truth). - formatters._normalize_to_ai() stamps schema_version on every AI-normalized output (ok path, error path, non-dict path). - formatters.format_output() also stamps raw JSON output (on a copy, so the caller's dict is not mutated). - Idempotent: if schema_version is already present, it is preserved. - Downstream agents can read this field to detect breaking changes and migrate gracefully. Tests: 54 new tests in tests/test_confidence.py covering score_finding, enrich_finding, enrich_findings, confidence_distribution, schema_version stamping, dead-code/secrets engine integration, and formatter integration (AI/JSON/markdown paths). No regressions in safe-to-run test subset (442 passed, 33 skipped, 0 failed). Pre-existing failures in test_universal_grammar_loader (tree-sitter-zig env) and test_codelensignore are unchanged by this PR. Sub-features #2 (delta output), #3 (dependency graph), #4 (partial failures) deferred to follow-up issues — they are larger and benefit from the confidence + schema_version foundation landing first.
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
This was referenced Jul 2, 2026
Wolfvin
added a commit
that referenced
this pull request
Jul 3, 2026
…#159) Upgrade `codelens secrets` to use gitleaks as the primary backend when available (600+ maintained rules, entropy scoring, structured JSON output). Falls back to the existing regex scanner when gitleaks is not installed or `--no-gitleaks` is set. Gitleaks is an opt-in upgrade, never a hard dependency. New files: - scripts/gitleaks_backend.py — self-contained gitleaks integration (454 lines) - _gitleaks_available() — detects binary via `gitleaks version` - _run_gitleaks() — invokes `gitleaks detect --no-git --report-format json`, writes to temp file, parses JSON, handles both list and {Results:[...]} schemas (older gitleaks versions), cleans up temp file - _normalize_gitleaks_findings() — maps gitleaks JSON fields to CodeLens finding schema (RuleID→type, File→file, StartLine→line, Secret→masked match/value, Tags→tags, Fingerprint→fingerprint) - _infer_severity() — heuristic severity mapping (aws-access-key/private-key/ github-pat/stripe-secret/etc → critical; 'critical'/'high'/'medium'/'low' in rule ID or tags → matching severity; default → high) - _mask_secret() — first 4 chars + *** (matches secrets_engine convention) - scan_with_gitleaks() — public entry point; returns CodeLens-shaped result dict with backend='gitleaks', or None if gitleaks unavailable - GitleaksError — caught by caller, falls back to regex with warning - tests/test_secrets_gitleaks.py — 56 tests (543 lines) - Detection (mocked subprocess: available, not-found, timeout, nonzero, unexpected exception) - Masking (long, short, empty, 4-char, 5-char) - Severity inference (aws/private-key/github-pat/stripe → critical; tags with high/medium/low; default → high) - Normalization (basic, masking, abs→rel paths, tags as string, empty, non-dict entries, missing fields) - Stats/risk/recommendations (counts, files_with_findings, critical→high →medium→low risk cascade, recommendation content) - _run_gitleaks (mocked: JSON parse, empty file, missing file, nonzero exit, timeout, invalid JSON, {Results:[...]} schema) - scan_with_gitleaks (full integration mocked: result dict shape, severity filter, nonexistent workspace) - CLI integration (subprocess): --no-gitleaks in help, regex backend + install hint when gitleaks absent, --no-gitleaks suppresses hint, stats.backend field set Modified files: - scripts/commands/secrets.py (73 lines): - Add --no-gitleaks flag (force regex backend) - execute() tries gitleaks first (unless --no-gitleaks); on gitleaks failure, falls back to regex with stderr warning (never crashes) - Tags result with backend='regex' or 'gitleaks' - When gitleaks unavailable, adds gitleaks_hint field with install URL - stats.backend also set so compact/ai formatters pick it up - SKILL.md — add 'Gitleaks-Backed Secrets Scanner' to v8.2 capability pillars with install instructions Design decisions: - gitleaks logic in NEW module (gitleaks_backend.py), not in secrets_engine.py — avoids collision with PR #148 (which touches secrets_engine.py for confidence scores). commands/secrets.py is the wiring point. - Gitleaks is OPT-IN: if not installed, regex runs unchanged with a hint. Never a hard dependency. - --no-gitleaks forces regex even if gitleaks is available — useful for testing, offline environments, or when gitleaks produces unexpected results. - --no-git flag passed to gitleaks (scan working tree, not git history) to match the regex backend's behavior. Git history scanning can be added later via a --git-history flag if needed. - Normalized findings include 'backend' field so consumers can tell which scanner produced each finding (useful when mixing backends in future). Verified: - 56 new tests pass (test_secrets_gitleaks.py — all subprocess mocked) - 109 passed regression check (test_secrets_engine + test_cli + test_command_count + test_command_registry — no regressions) - Command count unchanged at 70 (same command, upgraded backend) - End-to-end: --no-gitleaks in help, regex backend + hint when gitleaks absent, --no-gitleaks suppresses hint, stats.backend set Findings (per pre-flight SKILL.md — flag to BOS): - PR #148 (issue #5, open) touches scripts/secrets_engine.py (+16 lines, adding confidence scores). This PR touches scripts/commands/secrets.py (different file). No collision. Both can merge independently. The gitleaks backend's normalized findings include a 'confidence' field position in the schema — PR #148's confidence.py module can score gitleaks findings the same way as regex findings if desired.
Wolfvin
added a commit
that referenced
this pull request
Jul 3, 2026
…#159) Upgrade `codelens secrets` to use gitleaks as the primary backend when available (600+ maintained rules, entropy scoring, structured JSON output). Falls back to the existing regex scanner when gitleaks is not installed or `--no-gitleaks` is set. Gitleaks is an opt-in upgrade, never a hard dependency. New files: - scripts/gitleaks_backend.py — self-contained gitleaks integration (454 lines) - _gitleaks_available() — detects binary via `gitleaks version` - _run_gitleaks() — invokes `gitleaks detect --no-git --report-format json`, writes to temp file, parses JSON, handles both list and {Results:[...]} schemas (older gitleaks versions), cleans up temp file - _normalize_gitleaks_findings() — maps gitleaks JSON fields to CodeLens finding schema (RuleID→type, File→file, StartLine→line, Secret→masked match/value, Tags→tags, Fingerprint→fingerprint) - _infer_severity() — heuristic severity mapping (aws-access-key/private-key/ github-pat/stripe-secret/etc → critical; 'critical'/'high'/'medium'/'low' in rule ID or tags → matching severity; default → high) - _mask_secret() — first 4 chars + *** (matches secrets_engine convention) - scan_with_gitleaks() — public entry point; returns CodeLens-shaped result dict with backend='gitleaks', or None if gitleaks unavailable - GitleaksError — caught by caller, falls back to regex with warning - tests/test_secrets_gitleaks.py — 56 tests (543 lines) - Detection (mocked subprocess: available, not-found, timeout, nonzero, unexpected exception) - Masking (long, short, empty, 4-char, 5-char) - Severity inference (aws/private-key/github-pat/stripe → critical; tags with high/medium/low; default → high) - Normalization (basic, masking, abs→rel paths, tags as string, empty, non-dict entries, missing fields) - Stats/risk/recommendations (counts, files_with_findings, critical→high →medium→low risk cascade, recommendation content) - _run_gitleaks (mocked: JSON parse, empty file, missing file, nonzero exit, timeout, invalid JSON, {Results:[...]} schema) - scan_with_gitleaks (full integration mocked: result dict shape, severity filter, nonexistent workspace) - CLI integration (subprocess): --no-gitleaks in help, regex backend + install hint when gitleaks absent, --no-gitleaks suppresses hint, stats.backend field set Modified files: - scripts/commands/secrets.py (73 lines): - Add --no-gitleaks flag (force regex backend) - execute() tries gitleaks first (unless --no-gitleaks); on gitleaks failure, falls back to regex with stderr warning (never crashes) - Tags result with backend='regex' or 'gitleaks' - When gitleaks unavailable, adds gitleaks_hint field with install URL - stats.backend also set so compact/ai formatters pick it up - SKILL.md — add 'Gitleaks-Backed Secrets Scanner' to v8.2 capability pillars with install instructions Design decisions: - gitleaks logic in NEW module (gitleaks_backend.py), not in secrets_engine.py — avoids collision with PR #148 (which touches secrets_engine.py for confidence scores). commands/secrets.py is the wiring point. - Gitleaks is OPT-IN: if not installed, regex runs unchanged with a hint. Never a hard dependency. - --no-gitleaks forces regex even if gitleaks is available — useful for testing, offline environments, or when gitleaks produces unexpected results. - --no-git flag passed to gitleaks (scan working tree, not git history) to match the regex backend's behavior. Git history scanning can be added later via a --git-history flag if needed. - Normalized findings include 'backend' field so consumers can tell which scanner produced each finding (useful when mixing backends in future). Verified: - 56 new tests pass (test_secrets_gitleaks.py — all subprocess mocked) - 109 passed regression check (test_secrets_engine + test_cli + test_command_count + test_command_registry — no regressions) - Command count unchanged at 70 (same command, upgraded backend) - End-to-end: --no-gitleaks in help, regex backend + hint when gitleaks absent, --no-gitleaks suppresses hint, stats.backend set Findings (per pre-flight SKILL.md — flag to BOS): - PR #148 (issue #5, open) touches scripts/secrets_engine.py (+16 lines, adding confidence scores). This PR touches scripts/commands/secrets.py (different file). No collision. Both can merge independently. The gitleaks backend's normalized findings include a 'confidence' field position in the schema — PR #148's confidence.py module can score gitleaks findings the same way as regex findings if desired.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #5 (subset: sub-features #1 + #5)
Summary
Two agent-UX features from issue #5, implemented as a non-breaking enhancement so existing consumers keep working. Sub-features #2 (delta output), #3 (dependency graph), #4 (partial failures) are deferred to follow-up — they are larger and benefit from this foundation landing first.
What changed
1. Confidence scores per finding (sub-feature #1)
New file:
scripts/confidence.pyscore_finding(engine, category, finding)→ float in [0.0, 1.0]enrich_findings(engine, payload)enriches both dead-code (category-keyed dict) and secrets (flat list) payload shapesconfidence_distribution(findings)buckets into high/medium/low/unscored for quick reliability summariesModified:
scripts/deadcode_engine.pydetect_dead_code()callsenrich_findings("dead_code", payload)before returning — every finding now has aconfidencefieldModified:
scripts/secrets_engine.pydetect_secrets()callsenrich_findings("secrets", payload)before returning — every finding now has aconfidencefield2. Schema versioning (sub-feature #5)
SCHEMA_VERSION = "8.2"inscripts/confidence.py(single source of truth)stamp_schema_version(payload)adds the field idempotently (preserves existing value)formatters._normalize_to_ai()stamps every AI-normalized output (ok / error / non-dict paths)formatters.format_output()also stamps raw JSON output (on a copy, so caller's dict is not mutated)schema_versionto detect breaking changes and migrate gracefullyExample output
Before:
{ "status": "ok", "findings": [ {"type": "pattern_match", "file": "config.js", "line": 5, "category": "api_key"} ] }After:
{ "status": "ok", "schema_version": "8.2", "findings": [ {"type": "pattern_match", "file": "config.js", "line": 5, "category": "api_key", "confidence": 0.9} ] }Test results
tests/test_confidence.pypass:TestSchemaVersion(5) — stamping idempotency, non-dict handlingTestScoreFinding(13) — base scores, modifiers, range safety, fail-safeTestEnrichFinding(6) — idempotency, secrets vs dead-code field resolutionTestEnrichFindings(6) — both payload shapes, idempotency, preservationTestConfidenceDistribution(6) — bucketing, boundaries, non-dict handlingTestDeadCodeEngineIntegration(3) — real engine returns confidence fieldsTestSecretsEngineIntegration(3) — real engine returns confidence fieldsTestFormatterIntegration(7) — schema_version in AI/JSON/markdown pathsTestEndToEndSchemaVersion(4) — full formatter pipelinetest_universal_grammar_loader(tree-sitter-zig env) andtest_codelensignoreare unchanged by this PR (verified viagit stash).Non-breaking
confidencefield is additive — existing consumers ignore unknown fields.schema_versionfield is additive — same.sync_command_count.py --checkpasses.confidence.pyis missing.Deferred to follow-up
--since-lastflag)--correlateflag){skipped, reason, rest})These benefit from the confidence + schema_version foundation landing first.