Repository navigation
Time for the temporal test — do my metrics move with labeled commits? #2998
squid-protocol
announced in
Build Process Journal
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
I just finished two big improvement cycles, so I finally get to point the engine at the
question it was really built to answer: do my measurements move with a repo's actual
history? Concretely —
I had a set of 13 equations I'd been calling "risk exposure." The honest version of the
question: do the risk-exposure scores predict bugs, or drop when bugs get fixed?
How it's set up
This wasn't a vibe check — I pre-registered every prediction before looking at the data, in a
hypothesis ledger (
docs/HYPOTHESES.md, mirrored on gitgalaxy#2982). Direction, test, andα written down first; verdict published either way, no matter how it turned out. Three of
my first five predictions died. That's the ledger doing its job.
Ground truth so far:
ranges, plus 1,107 bug-label events (
Fixes #…,Bug:, regressions).Negative controls throughout: random commits and doc-only commits as negatives, plus
size-matched controls and length-matched function pairs, and temporal ablation so churn/
stability can't predict their own events.
The honest result: "risk exposure" was the wrong name
The risk-exposure metrics don't correlate with fix resolution. Fixes don't lower them
(H1: p=0.77), introducing commits don't reliably raise them (H2: p=0.023 against an α of
0.01), and they don't beat a plain line count at ordering a review budget (p=0.38).
That was enough to make me rename the whole family: these are surface-area /
structural-exposure metrics, not risk. They measure how much surface a file exposes —
which is real, and useful for describing and localizing code across 59 languages — but
surface area isn't risk, and I shouldn't have called it that. (The rename's now shipping in
the engine as the "Structural Surface Profile" vocabulary.)
The false positives, and the repowise result that explained them
I got a ton of false positives, and they all pointed the same way: anything that scales with
length looks correlated with bugs, because bigger files just have more bugs. This is exactly
the size-confound wall repowise-bench wrote about — genuinely the clearest breakdown of
this problem I've seen, worth reading. I ran the same check on CVE labels and hit the same
wall: pooled, density-normalized, length-banded, sliced every which way — nothing
structural beats a plain line count (nulls equivalence-confirmed on both repos).
Once I had that lens, two things actually survived it — and both are about history, not
structure:
ranking on both repos (p<1e-4). Strongest, most reproducible result by far.
selected on CVE labels, then validated cold on fresh bug labels and still beat line
count (AUC 0.868 vs 0.830). The only structural-ish feature to survive out-of-selection.
Scoreboard so far
Died (evidence against):
Fixes lowering aggregate structural exposure — nope (p=0.77). The idea was that once
you patch a vulnerable file, its overall exposure score should drop. It doesn't, at all.
The pre-fix and post-fix scores are statistically indistinguishable.
Introducing commits raising it — nope, at my threshold. The mirror-image hypothesis:
commits that introduce a bug or CVE should push exposure up. There's a whisper of a
signal (p=0.023), but I'd set my α at 0.01 specifically so I couldn't talk myself into a
marginal result later. It doesn't clear the bar I set for myself before looking.
Structural exposure beating LOC for ranking review budget — nope (p=0.38). If you had
a fixed amount of reviewer time and had to decide which files to look at first, does my
metric out-rank "just sort by file size"? No — they perform statistically the same, which
means the metric isn't adding any ranking value over the dumbest possible baseline.
Danger-density flagging vulnerable functions at equal length — nope, fell apart on nDPI
(p=0.47). This one controlled for size directly: take functions of the same length and
see if the "denser" ones are more likely to be the vulnerable one. Looked plausible on
curl, then completely failed to replicate on nDPI. That non-replication is why it's in the
"died" column and not the "survived" one.
Under-guarding of vulnerable functions as a signal — nope, degenerate at function
granularity. The theory was that vulnerable functions would show measurably less
defensive guarding (null checks, bounds checks, etc.) than safe ones. At the function
level the metric basically stops discriminating at all — too noisy to be usable, not just
"didn't show an effect."
The "CVE grammar" (branch/pointer adds without new allocations) as a general law —
partially real, but not a law. This is a specific commit-shape pattern: fixes that add
branches or pointer checks without adding new memory allocations. It's genuinely there in
curl and survives controls for fix size. But it goes completely silent on nDPI's
fuzzer-found fixes — zero signal. So it looks like a property of how people report and
fix CVEs in curl specifically, not a general rule about how security fixes look.
Survived (evidence for):
Recidivism beats every static ranking (p<1e-4, both repos). "The file that got fixed
last time is the most likely one to need fixing next" turned out to be the single strongest
predictor in the whole study — stronger than anything structural, and it held up
independently on both curl and nDPI, which is a much higher bar than looking good on one
repo alone.
Change entropy beats line count out-of-selection (0.868 vs 0.830 AUC, validated
frozen). Change entropy (Hassan's HCM — a measure of how scattered and chaotic a file's
edit history is) was picked as a promising feature using the CVE data, then locked and
tested cold against a completely separate set of bug labels it had never seen. It still
beat plain line count. That "frozen validation" step is what makes this result trustworthy
rather than just a good fit to the data it was chosen on. Shout out to repowise on this, they tried 1 category but Hassan had 36 and at least for this small set, that was the more correlative one, which is different than Repowise. I'm curious to see if this preliminary finding holds up.
Commit classes have distinct structural fingerprints in the keyword vocabulary. Even
though the aggregate exposure score is useless for prediction, individual commit types
(fix commits vs. introducing commits vs. everything else) show clearly different patterns
in the underlying keyword-level features. In other words, the instrument is reading real
security-relevant signal — it's just getting lost when you collapse everything into one
summary number. My read on why: fixes tend to add guards and remove debt-density at the
same time, and those two effects point in opposite directions, so they cancel out in the
aggregate. The signal isn't gone, it's just hiding at a level of detail the aggregate score
throws away. More exploration is needed!
Bonus: chasing all this down also caught a real determinism bug in the engine (gitgalaxy#2988).
More predictions died than lived, which is honestly the point.
The big caveat, no sugarcoating
Two repos, both C-family, ~3,550 snapshots total — not a lot of ecosystem diversity. Recidivism
held up on both curl and nDPI, but literally every structural metric that looked promising on
curl fell apart on nDPI. So right now I can say "recidivism is real, structural exposure is
just a size proxy in disguise" — but only for C. No cross-language claim yet. That's exactly
why repo #3 is openssl: see if the grammar shows back up on human-reported fixes elsewhere,
see if entropy replicates again, and finally run the CWE-matched specificity battery that
curl and nDPI were both too underpowered for.
More as it comes in.
All reactions