bug: v0.8.1 regressed the held-out suite and shipped without re-recording, so the site publishes accuracy it no longer achieves

[aryamthecodebreaker]

Measured against main@39d6a1d with a clean npm ci and a rebuilt core, on the same machine and the same cached checkouts used for the v0.8.0 baseline.

The held-out suite regressed

metric	v0.8.0 (measured)	v0.8.1 (measured)	committed results.json
top-1	7/12 · 0.583	6/12 · 0.500	7/12 · 0.583
top-3	9/12 · 0.750	7/12 · 0.583	9/12 · 0.750
top-5	9/12 · 0.750	8/12 · 0.667	9/12 · 0.750

benchmarks/heldout/results.json was never refreshed, so it still records the v0.8.0 result. The other suites: internal rose 0.625/0.875 → 0.875/1.000, external unchanged at 0.667/1.000/1.000, adversarial gate still green.

Dev suite up, generalization suite down is the shape the held-out set exists to detect.

This is currently a false public claim

apps/web/app/_lib/site-data.ts derives the headline metric from that JSON at build time, so https://usefixmap.vercel.app/evidence and the home page are both serving "9/12 held-out fixes surfaced in the top three" while the published 0.8.1 produces 7/12. The site is advertising an accuracy figure the released code does not achieve.

That is the same class of problem as #86, where a recorded result had drifted from what the committed ranker produced and the README published the stale figure.

Which cases moved, and why

Automattic/mongoose   1/1 -> 1/1
immerjs/immer         1/1 -> 1/1
jestjs/jest           0/1 -> 0/0   <-- CHANGED
knex/knex             1/1 -> 1/1
mochajs/mocha         0/1 -> 0/1
sindresorhus/got      1/1 -> 0/0   <-- CHANGED
socketio/socket.io    0/0 -> 0/0
sveltejs/svelte       1/1 -> 1/1
vitejs/vite           0/0 -> 0/0
vuejs/core            0/0 -> 0/0
winstonjs/winston     1/1 -> 1/1
yargs/yargs           1/1 -> 1/1

Two different causes.

got — #243's URL stripping removed a real signal, not noise

source/core/index.ts went from rank 1 to below the reporting cutoff:

Scored 5, below the lowest reported score of 14.
Signals: path matches task terms: source

Its entire content signal is gone because the file is 79,054 bytes, over MAX_TEXT_SAMPLE_BYTES (64,000) — true in both versions, so the sample was always empty. In v0.8.0 the file ranked first purely on EXPLICIT_PATH_BOOST, and that boost came from the issue body:

https://github.com/sindresorhus/got/blob/e5e645a.../source/core/index.ts#L1088-L1089
https://github.com/sindresorhus/got/blob/e5e645a.../source/core/index.ts#L1093

#243 now strips every URL before extractFileMentions runs, so the mention disappears.

#243 was right about raw.githubusercontent.com badges and links to unrelated repositories. But a GitHub blob permalink pointing into the repository being scanned, with a line range, is the single most deliberate "the bug is here" signal a human can leave in an issue. Stripping all URLs discards it along with the noise.

Suggested narrowing: strip URLs generally, but first extract the path from github.com/<owner>/<repo>/blob/<ref>/<path> when <owner>/<repo> matches the repository being scanned. FixMap already parses GitHub URLs for --issue, so the machinery exists.

Note this also means the v0.8.0 held-out figure was partly propped up by the bug #243 fixed — worth stating plainly wherever the number is published, rather than presenting 0.75 as a clean prior result.

jest — a four-way tie decided by path sort order

packages/jest-mock/src/index.ts is still found, at rank 4:

1  medium  24  packages/jest-config/src/readConfigFileAndSetRootDir.ts
2  medium  24  packages/jest-config/src/resolveConfigPath.ts
3  medium  24  packages/jest-core/src/TestScheduler.ts
4  medium  24  packages/jest-mock/src/index.ts

All four score 24, and the tie-break is a.path.localeCompare(b.path), so jest-mock sorts last among equals and falls out of top-3 by alphabetical accident. This case's hit/miss is not currently measuring ranking quality.

What I think should happen

1. Re-record all suites and correct the published figure, or hold 0.8.1 until the regression is addressed. The site should not claim 9/12 while the release delivers 7/12.
2. Narrow [signals] raw.githubusercontent.com / URL paths can surface as file mentions #243 so same-repository blob permalinks still yield a file mention.
3. Treat the jest case as evidence that near-ties at the reporting boundary need a better tie-break than path order.

Per the project's own rule, if a ranking change is made because a held-out case missed, that case should move to external/ and a fresh replacement selected — otherwise the suite quietly stops measuring generalization.

Environment

Published 0.8.1, Node 24.13, Windows. Checkouts verified complete before measuring: sindresorhus__got-28c0ca3c6571 contains source/core/index.ts (120 files), so this is not the partial-clone artifact from #73.