Skip to content

godaudits 2.10.0

Choose a tag to compare

@hannsxpeter hannsxpeter released this 16 Jul 23:49
· 46 commits to main since this release
37ea52e

The detector gate now measures something real

2.9.0 built the measurement machinery and honestly reported nothing, because authored fixtures detect their own seeds by construction. This release supplies the missing input: recorded blind audit runs.

The experiment

Six seeded fixture repositories graded from blatant to genuinely subtle (an unscoped lookup of medical PII; a service whose list and update bind the tenant while the by-id read does not; a scoping helper one later handler bypasses; a handler that loads unscoped and filters afterward; an export ignoring the predicate its siblings apply) plus one control with no seeded defect at all.

Ground truth was authored before any audit ran and kept outside the repositories. Each auditor received only a repository path and the catalog's own A-SEC-3 definition: no ground truth, no defect count, no hint a control existed. Runs are captured verbatim, hits and misses alike.

5/5 seeded defects detected, 0 false positives on the control. A-SEC-3 clears the five-independent-audit floor and reports a measured detection rate of 1.0 with a Wilson lower bound of 0.5655 — five for five is not a perfect detector, and the artifact must not read as though it were. Checks carried only by authored fixtures still report authored-only and no rate.

The blind run corrected the ground truth

Ground truth for the post-filter case said High. The run said Critical and was right: A-SEC-3 keys severity on the data class ("Critical on PII, financial, or cross-tenant data"), and tickets across organizations are cross-tenant data. The original entry imported a compensating-control discount the rule does not contain, and had not weighed that the handler is fail-open when req.user is absent or that the 404/403 split is a cross-tenant existence oracle. Corrected in place, with the correction recorded rather than quietly rewritten.

Honest edges

Recorded findings carry Firm rather than the label the run reported, because the capture schema collected one citation each and one evidence path cannot support Certain: 2.8.0's corroboration invariant applied to real output. Recorded prose passes through the same secret redactor a real audit uses, which conservatively masked a shareToken: req.query.token code expression in remediation advice; the redactor was applied rather than loosened.

Scope

This measures detection on six fixtures, not a reliability estimate for unseen repositories, and no number here feeds a per-repo score. That boundary is enforced in code and stated in the artifact.

Tests 91 → 92; full gate green.