Summary
scripts/evaluate.mjs appends % to values that are fractions, so a failing evaluation gate reports rates ~100x smaller than reality.
const top3HitRate = hits / results.length; // 0..1
const top1HitRate = top1Hits / results.length; // 0..1
...
process.stderr.write(
`FixMap evaluation failed: top-1 ${top1HitRate.toFixed(1)}%, top-3 ${top3HitRate.toFixed(1)}%.\n`
);With 5 of 10 cases hitting top-1, the message reads top-1 0.5% instead of top-1 50.0%.
toFixed(1) on a fraction also collapses the useful precision — every rate below 5% renders as 0.0% and every rate from 5% to 15% renders as 0.1%.
Impact
This string is the only human-readable output when the CI gate fails, so the one moment it matters is the one moment it is wrong. A contributor reading top-1 0.4%, top-3 0.7% against documented thresholds of 0.5 / 0.8 cannot tell whether the ranker regressed slightly or collapsed.
The JSON summary written to stdout just above is correct (it reports the raw fractions with the matching thresholds object) — only the stderr line is wrong.
Suggested fix
const asPercent = (rate) => `${(rate * 100).toFixed(1)}%`;
process.stderr.write(
`FixMap evaluation failed: top-1 ${asPercent(top1HitRate)} (min ${asPercent(summary.thresholds.top1)}), ` +
`top-3 ${asPercent(top3HitRate)} (min ${asPercent(summary.thresholds.top3)}).\n`
);scripts/evaluate-external.mjs does not have this bug, but its failure message is "External evaluation fell below regression floors." with no numbers at all — worth giving both scripts the same "measured vs floor" formatting while touching this.
Summary
scripts/evaluate.mjsappends%to values that are fractions, so a failing evaluation gate reports rates ~100x smaller than reality.With 5 of 10 cases hitting top-1, the message reads
top-1 0.5%instead oftop-1 50.0%.toFixed(1)on a fraction also collapses the useful precision — every rate below 5% renders as0.0%and every rate from 5% to 15% renders as0.1%.Impact
This string is the only human-readable output when the CI gate fails, so the one moment it matters is the one moment it is wrong. A contributor reading
top-1 0.4%, top-3 0.7%against documented thresholds of0.5/0.8cannot tell whether the ranker regressed slightly or collapsed.The JSON summary written to stdout just above is correct (it reports the raw fractions with the matching
thresholdsobject) — only the stderr line is wrong.Suggested fix
scripts/evaluate-external.mjsdoes not have this bug, but its failure message is"External evaluation fell below regression floors."with no numbers at all — worth giving both scripts the same "measured vs floor" formatting while touching this.