Surfaced by
The eval harness from #65 (PR #149). The noPreambleLeak criterion in src/lib/webllm/eval/rubric.ts is reporting 100% pass on Llama runs that visibly leaked the preamble "Here are the rewritten bullets:" as the first bullet — that's a false positive.
Root cause
The current implementation (rubric.ts):
let rawMinusBullets = output.raw.toLowerCase();
for (const b of outputBullets) {
rawMinusBullets = rawMinusBullets.replace(b.toLowerCase(), "");
}
const noPreambleLeak = !PREAMBLE_LEAK_PHRASES.some((p) =>
rawMinusBullets.includes(p),
);We strip every output bullet from raw before scanning for preamble phrases. The intent was to avoid false positives when a legitimate bullet happens to contain the phrase (e.g. a bullet about "the rules of engagement"). But when the preamble itself becomes a bullet (because cleanRewriteLine didn't strip it — see #150), our own bullet-stripping step erases the leaked preamble from the scanned text. The check then passes vacuously.
Fix
Add a second check: any individual bullet whose text matches a preamble phrase should also fail noPreambleLeak. Pseudocode:
const anyBulletIsPreamble = outputBullets.some((b) =>
PREAMBLE_LEAK_PHRASES.some((p) => b.toLowerCase().includes(p)),
);
const noPreambleLeak = !anyBulletIsPreamble && !PREAMBLE_LEAK_PHRASES.some((p) =>
rawMinusBullets.includes(p),
);
This makes the rubric robust to both "preamble survived in raw text" and "preamble survived as a bullet" failure modes.
Add a unit test in rubric.test.ts that pins this: input with raw "Here are the rewritten bullets:\nDrove a 4-touchpoint nurture sequence..." should fail noPreambleLeak regardless of whether the preamble was line-split into a bullet or kept inline.
Tracking
Surfaced by
The eval harness from #65 (PR #149). The
noPreambleLeakcriterion insrc/lib/webllm/eval/rubric.tsis reporting 100% pass on Llama runs that visibly leaked the preamble"Here are the rewritten bullets:"as the first bullet — that's a false positive.Root cause
The current implementation (
rubric.ts):We strip every output bullet from
rawbefore scanning for preamble phrases. The intent was to avoid false positives when a legitimate bullet happens to contain the phrase (e.g. a bullet about "the rules of engagement"). But when the preamble itself becomes a bullet (becausecleanRewriteLinedidn't strip it — see #150), our own bullet-stripping step erases the leaked preamble from the scanned text. The check then passes vacuously.Fix
Add a second check: any individual bullet whose text matches a preamble phrase should also fail
noPreambleLeak. Pseudocode:This makes the rubric robust to both "preamble survived in raw text" and "preamble survived as a bullet" failure modes.
Add a unit test in
rubric.test.tsthat pins this: input with raw"Here are the rewritten bullets:\nDrove a 4-touchpoint nurture sequence..."should failnoPreambleLeakregardless of whether the preamble was line-split into a bullet or kept inline.Tracking
post-process.ts— once both land, this blind spot stops mattering in practice, but the rubric is still measuring the wrong thing today).