bug: stemmer fix regressed four-letter bases ending in e (base/based, file/filed, date/dated)

[aryamthecodebreaker]

Follow-up to #69, introduced by #82 (b1990da). The original defect is genuinely fixed — failed/fail, boxes/box, routing/route, parsing/parse all match now, and doubled consonants (stopped/stop) are handled. But the new rule breaks a different set of words, and five of them used to work.

I raised this on the PR before merge; recording it here so it is tracked rather than lost in a merged thread.

Verified on merged main

Comparing tokenizeText at 352ec03 (pre-merge) against b1990da (merged):

pair           pre-merge    merged       verdict
filed/file     MATCH        mismatch     REGRESSION
dated/date     MATCH        mismatch     REGRESSION
sized/size     MATCH        mismatch     REGRESSION
timed/time     MATCH        mismatch     REGRESSION
based/base     MATCH        mismatch     REGRESSION
coding/code    mismatch     mismatch     still broken

The affected class is four-letter bases ending in e: base, code, file, size, date, time, line, mode, page, case, role, site, rule. That is close to a list of the most common nouns in a repository-routing tool's task vocabulary, and based in particular shows up in ordinary issue prose ("based on the stack trace...").

Cause

normalizeTrailingE guards on length > 4, so a four-letter base keeps its e while its -ed/-ing form stems to a three-letter root with no e left to strip:

function normalizeTrailingE(token: string): string {
  return token.length > 4 && token.endsWith("e") ? token.slice(0, -1) : token;
}

file (4) is not > 4, so it stays file. filed -> -ed -> normalizeVerbStem("fil") -> fil. No match.

Candidate fix and its cost

Relaxing the guard to length > 3 makes all six pairs above match — I verified this directly against the merged source. It is not free, which is why this is an issue rather than a patch:

1. Stop-word collisions. note -> not and here -> her, both in STOP_WORDS, so those tokens get dropped entirely. note is unfortunate given RiskNote / "risk notes" is FixMap's own vocabulary. (core, page, role, line are fine — they become cor, pag, rol, lin.)
2. The -s rule returns early, skipping normalizeTrailingE:

   if (token.length > 3 && token.endsWith("s")) return token.slice(0, -1);

   Under a > 3 guard this makes notes -> note but note -> not, so singular/plural stops matching. That rule needs to route through normalizeTrailingE too.

Suggested test shape

The reason both this regression and the original #69 slipped through is that signals.test.ts asserts literal token strings for a hand-picked pair of words, so it only ever covered cases that happened to work. A round-trip assertion over a table would catch the whole class:

const INFLECTIONS = [
  ["failed", "fail"], ["boxes", "box"], ["routing", "route"], ["parsing", "parse"],
  ["created", "create"], ["invoices", "invoice"], ["stopped", "stop"], ["cached", "cache"],
  ["filed", "file"], ["dated", "date"], ["sized", "size"], ["timed", "time"],
  ["based", "base"], ["coding", "code"]
];

it.each(INFLECTIONS)("stems %s and %s identically", (inflected, base) => {
  expect([...tokenizeText(inflected)]).toEqual([...tokenizeText(base)]);
});

Asserting that the two stem to the same thing rather than to a specific string keeps the -ed, -ing, -es, -s, and trailing-e rules from disagreeing again, without pinning the exact stem.

Whichever direction is taken, npm run evaluate is the arbiter — current numbers on main are top-1 62.5%, top-3 87.5% over 8 cases.