Scorecard/code-scanning findings recur estate-wide: Hypatia never closes the alert-lifecycle loop

[hyperpolymath]

Summary

OSSF Scorecard / code-scanning findings recur on every audit, estate-wide, and have for months. Concretely, modshells has 3 open alerts (Pinned-Dependencies #63, SAST #72, Maintained #44) that Hypatia has never closed. Root-causing them exposes a structural defect: Hypatia ingests findings into its own pipeline but never closes the loop on the authoritative GitHub alert lifecycle. It re-derives findings from local heuristics, fixes a subset, and never (a) reads the real open alerts, (b) records accepted exceptions, (c) verifies tool efficacy vs mere presence, or (d) dismisses non-actionable findings with a reason. So the same three classes leak forever.

This issue tracks the Targeted fixes + Scorecard↔GitHub reconciliation loop, plus a no-holds-barred design refinement so Hypatia's logic actually prevails.

Refs — joint-close only on explicit agreement (do not auto-close).

---

What Hypatia is supposed to do — and keeps failing to

Intended loop: scan → classify (safety triangle: eliminate/control/accept) → auto-fix where safe → learn → keep the estate continuously green.

Observed loop: scan (local heuristics) → classify → fix-a-subset. Missing: authoritative sense, verify, and learn-write-back. That open loop is the disease. Three leak classes:

• (A) Non-actionable (Maintained, Contributors): no code fix exists. Must be dismissed with reason, not re-discovered every run. → alert chore(deps): bump trufflesecurity/trufflehog from 3.92.5 to 3.93.0 #44.
• (B) Design-correct / accepted exception (SLSA generator must stay on a semver tag — SHA-ref produces invalid provenance): Hypatia's only symbolic remediation (SHA-pin) is actively harmful, with no exception channel. → alert chore(deps): bump ruby/setup-ruby from 1.288.0 to 1.289.0 #63.
• (C) Nominal-not-effective (CodeQL present but pointed at a language the repo doesn't contain → 0 results): presence check passes, tool produces nothing. → alert chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72.

Root-cause defects in the symbolic engine

1. Dark rule. WorkflowAudit.check_codeql_language_matrix_mismatch already encodes the correct fix (:switch_codeql_matrix_to_actions) but is gated Keyword.get(opts, :has_codeql_supported_language, true) — defaults true → silent no-op unless a caller computes language detection (it never does). (chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72)
2. Open-loop ingestion. ScorecardIngestor maps checks into Hypatia's pipeline but there is no write path back to GitHub's code-scanning alert state (no dismiss/accept). Non-fixable + exception findings re-accumulate as "open" every audit. (meta / chore(deps): bump trufflesecurity/trufflehog from 3.92.5 to 3.93.0 #44 / chore(deps): bump ruby/setup-ruby from 1.288.0 to 1.289.0 #63)
3. Presence ≠ efficacy. ScorecardIngestor.check_sast greps for the string "codeql". modshells has codeql.yml, so Hypatia believes SAST is satisfied while Scorecard reports 0/7 commits checked. (chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72)
4. Harmful canonical remediation, no exception. workflow_audit.ex:49 and security_errors.ex:291 map slsa-framework/slsa-github-generator@v2.1.0 → <sha>. Applying it breaks SLSA provenance. No exemption registry exists. (chore(deps): bump ruby/setup-ruby from 1.288.0 to 1.289.0 #63)
5. No suppression memory. Nothing persists "this finding on this repo is accepted because X," keyed by a stable fingerprint, so anything dismissed re-fires next scan. (the recurrence engine itself)

---

Part 1 — Targeted fixes (sub-issues)

• [ ] CodeQL efficacy + un-dark the matrix rule. Default has_codeql_supported_language to computed-from-linguist, not true. Make absence of a scannable language the trigger. Add a paired effective-SAST rule: finding when codeql.yml's language matrix ∌ any language the repo contains and ∌ actions. Recommend language: actions (always scannable; runs every commit).
• [ ] SLSA / self-verifying-ref exemption registry. Remove the harmful SHA mapping for slsa-framework/slsa-github-generator. Introduce a first-class pin_exempt / must_track_tag registry (data, not code constants) for reusable workflows that self-verify github.ref. Emit an accept finding with rationale, never a fix.
• [ ] Hypatia.ScorecardReconciler (the closer). Pull live code-scanning alerts via the GitHub API. For each: actionable → dispatch fix; accepted-exception → dismiss won't fix + rationale comment; non-actionable/informational → dismiss won't fix + rationale. Idempotent. Every dismissal recorded by stable (repo, rule, location_fingerprint) in a persisted exceptions store so re-scans do not re-open.

Part 2 — Neurosymbolic anti-recurrence (no holds barred)

Symbolic

• Triage taxonomy generalization. Replace per-check booleans with a 4-axis classifier: {actionable_by_code?, remediation_is_safe?, effective_vs_nominal?, activity_only?}. Every Scorecard check and code-scanning rule maps to one cell; the cell deterministically selects the lifecycle action (fix / dismiss-accept / dismiss-info / open-and-alert). General (new rules slot in) and narrow (one unambiguous action per cell).
• Detector splitting (sensitivity ⟂ specificity). For every "tool present" rule, add a paired "tool produced results in last N commits" rule (query analyses / check-runs API). Presence = specificity; efficacy = sensitivity. This is the generalisation of fix chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72.
• Exception/known-good data, not code constants. Move @known_good_shas + exemptions into versioned data the learning loop can write.
• Fingerprint-stable suppressions. Key everything on repo+rule+normalized-path+symbol, never line number (lines drift; fingerprints don't). This single change is the recurrence killer.

Neural

• LLM exception adjudicator. When a finding's symbolic cell is ambiguous (e.g., a new third-party reusable workflow that might self-verify its ref), an LLM judges "is SHA-pinning safe or harmful?" using the alert help text + upstream README, proposing a lifecycle action + rationale. Neural proposes; the decision is crystallised into the symbolic exception registry and is symbolic thereafter.
• Remediation-safety predictor. Small classifier over (action, fix-type, downstream-effect) trained on past outcomes (did the fix break CI? did the alert reopen?) to gate auto-fix vs accept.
• Recurrence forecaster (the months-long-pain killer). Cluster findings by (rule, repo-class, root-cause-signature). When a cluster's reopen-rate exceeds threshold, escalate from per-repo fix to a template fix (PR into rsr-template-repo / v3-templater). The meta-signal is reopen-rate; the meta-action is template propagation. Attack the source, not the leaves.

Part 3 — Meta-layers, connections, data model (logic must prevail)

• Closed-loop control architecture. Sense(authoritative = live GitHub alerts API) → Classify(taxonomy + neural adjudicator) → Act(fix | dismiss-with-reason | escalate-to-template) → Verify(re-query alert state next run) → Learn(write exception/outcome; update reopen-rate). Today only Sense(local) → Classify → Act(subset) exists. Add authoritative-Sense, Verify, Learn-write-back.
• Normalized data model.
  • finding(id, repo, source, rule, location_fingerprint, first_seen, last_seen, state)
  • lifecycle_decision(finding_fp, action, rationale, decided_by[symbolic|neural|human], decided_at, confidence)
  • exception_registry(scope[repo|estate], rule, predicate, rationale, expiry?) — versioned, machine+human writable
  • outcome(finding_fp, fix_pr, ci_result, reopened?, reopened_after_days) — feeds the forecaster
• Connections. gitbot-fleet must carry lifecycle decisions + exceptions, not just raw findings, so one adjudicated SLSA exception auto-applies to all ~30 estate repos sharing that workflow fingerprint. Hypatia ↔ rsr-template-repo/v3-templater: forecaster opens template PRs. Event-driven (repository_dispatch on a new Scorecard run), not only weekly cron → responsiveness.
• Logic-prevails invariants. Every neural decision must (a) reduce to a symbolic registry entry before enforcement, (b) be reversible, (c) carry a rationale string, (d) be confidence-gated with human escalation below threshold. Neural may propose and prioritise; only symbolic rules + registry may enforce.

---

Reactive companion (already in flight)

modshells: codeql.yml matrix javascript-typescript → actions (PR, Refs this issue); alerts #63 and #44 dismissed with written rationale (SLSA-tag-by-design; informational-activity-signal respectively).

Part 4 — Autonomy & credit-economics (the real KPI)

The binding constraint is the maintainer spends Claude credit every week manually getting Hypatia to (a) do its job and (b) learn anything. Success is therefore economic, not just green checks:

• North-star KPI: human/LLM interventions per estate-green-week → ~0; finding reopen-rate trending monotonically to 0; credit spend on estate maintenance flat or falling week-over-week. These are dashboardable from the outcome table.
• Learning must be cheap and symbolic-by-default. Every adjudication (neural or human) crystallises into the exception_registry so it is never re-reasoned. The expensive LLM path runs only on genuinely novel, ambiguous findings — bounded, logged, and amortised to a one-time cost per pattern. A finding class adjudicated once must never burn credit again.
• Self-healing without a human in the weekly loop. The closed loop (Sense→Classify→Act→Verify→Learn) runs on repository_dispatch/cron entirely in Actions; humans are paged only on (i) below-confidence novel findings or (ii) a fix that broke CI. "World ends, only Actions survives" ⇒ the loop must need no local machine, no interactive Claude, no .git-private-farm online — it degrades to symbolic-registry-only and still keeps every repo green.
• .git-private-farm role: durable, offline-survivable mirror of the exception_registry + outcome history so learning is not lost if GitHub state is reset — the "rebuild from DNA" substrate.

Part 5 — CI/CD adequacy & anti-redundancy audit

Goal: 100% CI success every run, covering corrective + adaptive + perfective maintenance, with no pointless/redundant workflows (genuine triangulation is kept).

• Coverage check: map the standard workflow set against {build, test, lint/format, SAST, secret-scan, dependency-update, supply-chain/provenance, license/SPDX, compliance/standards-drift, dead-config detection}. Flag any axis with zero effective (not merely nominal) coverage — the chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72 lesson generalised: a workflow that runs but verifies nothing is a coverage gap, not coverage.
• Redundancy/irrelevance check: identify workflows that are dead config for the repo's actual languages (e.g., a JS/TS CodeQL on an Ada repo, npm/bun blockers where no JS exists as enforcement vs as guardrail — keep guardrails, cut dead analyzers), duplicate scanners with no triangulation value, and workflows whose failure can never gate anything. Distinguish triangulation (keep) from redundancy (cut): triangulation = independent methods confirming the same property; redundancy = same method, no added assurance.
• Deliverable: a per-repo CI adequacy scorecard Hypatia computes itself (new rule class), feeding the same reconciliation loop so "inadequate or wasteful CI" becomes an auto-tracked, auto-PR'd finding like any other.

[hyperpolymath]

Part 6 — Six-Sigma KPI framework (so Bill Smith complains the bar is too high)

Success is economic and statistical, not "checks are green." Every KPI has a definition, a data source (the outcome/finding/lifecycle_decision tables), and a hard target. DPMO = defects per million opportunities; an "opportunity" = one (repo × check × scan).

A. Defect / quality (the Six-Sigma core)

1. Finding-recurrence DPMO — a defect = a finding that was previously closed/dismissed and re-opened with no underlying change. Target < 3.4 DPMO (6σ). Stretch: 0.
2. Harmful-remediation rate — fixes that broke CI or invalidated provenance (the SLSA class). Target 0 (zero-tolerance; any occurrence is a P0 rule defect, not a metric to "improve").
3. Misclassification rate — taxonomy cell ≠ correct lifecycle action, sampled/audited. Target < 233 DPMO (5σ) with monotonic improvement to 6σ.
4. False-dismissal rate — anything dismissed-with-reason that was actually actionable. Target 0; guarded by reversibility + audit sampling.
5. Escaped-defect rate — alerts open on main that Hypatia never even classified ("didn't look"). Target 0.

B. Autonomy / cost (the real pain)

6. Human/LLM interventions per estate-green-week. Target → 0; alarm if > 1.
7. Claude-credit spend on estate maintenance, week-over-week. Target monotonically non-increasing; each novel pattern is a one-time cost that must never recur (crystallised to registry).
8. Novel-finding rate per estate pass — fraction of findings with no matching registry entry. This is the KPI-gate for convergence. Target ≈ 0 over a full pass before the 50/100 streak starts.
9. Adjudication amortisation — count of registry entries reused ÷ entries created. Target strictly increasing (learning compounding).

C. Latency / responsiveness

10. Mean time to detect (commit → finding classified). Target < 1 scan cycle.
11. Mean time to remediate-or-dismiss (finding → closed/dismissed-with-reason). Target < 24h auto-path; < 1 cycle for registry-known classes.
12. Verify-loop closure rate — % of acted findings re-checked next cycle and confirmed closed. Target 100% (no fire-and-forget).

D. Coverage / capability (process capability, Cpk-style)

13. Effective coverage — axes {build,test,lint,SAST,secret,deps,provenance,license,compliance,dead-config} with effective (not nominal) coverage. Target 100%; nominal-only counts as a gap (the chore(deps): bump aquasecurity/trivy-action from 0.34.2 to 0.35.0 #72 lesson as a standing KPI).
14. CI green-rate at HEAD across estate. Target 100% every run (the stated goal).
15. Estate Cpk — treat per-repo compliance score as the process; require Cpk ≥ 2.0 (6σ-capable) before declaring convergence.

E. Durability ("rebuild from DNA")

16. Registry survivability — exception/outcome state fully reconstructable from .git-private-farm with GitHub state wiped. Target 100%, drill-tested.
17. Loop autonomy under degradation — with neural layer + local machine offline, symbolic-registry-only path still holds estate green. Target 100% of registry-known classes.

Governing rule: convergence (Phase 3) is declared only when KPIs 1–5 hit their σ targets and KPI 8 ≈ 0 over a full estate pass — then the 50-clean-then-100-confirm streak runs as the human-facing proof. KPIs 6–7 are the ones that protect your wallet and are reported every cycle.