fix: converge four-letter e-stems so base and inflected forms match

[aryamthecodebreaker]

Fixes #83.

#82 fixed the -ed/-es under-stripping reported in #69, but the new rule regressed a different class of words: four-letter bases ending in e. Five pairs that matched before #82 stopped matching.

Cause

normalizeTrailingE guarded on length > 4, so a four-letter base kept its e while its inflected form stemmed one character shorter:

function normalizeTrailingE(token: string): string {
  return token.length > 4 && token.endsWith("e") ? token.slice(0, -1) : token;
}

file (4) is not > 4, so it stays file. filed -> -ed -> normalizeVerbStem("fil") -> fil. No match.

Fix

One guard, > 4 -> > 3. The result is still three characters, which is the minimum token length tokenizeText keeps.

The -s path needs no change: a word ending in s but not es cannot leave a trailing e behind, so normalizeTrailingE would be a no-op there. files already routes through the -es rule.

Verified

Three-way against tokenizeText at 352ec03 (pre-#82), b1990da (merged), and this branch:

pair	pre-#82	merged	this branch
based/base	MATCH	mismatch	MATCH
filed/file	MATCH	mismatch	MATCH
dated/date	MATCH	mismatch	MATCH
sized/size	MATCH	mismatch	MATCH
timed/time	MATCH	mismatch	MATCH
coding/code	mismatch	mismatch	MATCH

Everything #82 fixed stays fixed — failed/fail, boxes/box, routing/route, parsing/parse, stopped/stop, cached/cache, created/create, invoices/invoice. 19 of 20 pairs in my check now converge; the one that does not is cases/case, which is unrelated and unchanged — case is itself in STOP_WORDS, so it is dropped before stemming.

Cost

note and here now stem onto the stop words not and her and are dropped entirely. That is the second stop-word filter in tokenizeText working as designed — the same mechanism that already drops does -> doe. It buys consistency for base, code, file, size, date, time, line, mode, page, and role, which is a much better trade for a repository-routing tool.

I considered keeping note by skipping the strip when the result is a stop word, but that leaves note/notes inconsistent with each other and adds a branch for two words, so I left it out.

Impact on ranking

None measurable. npm run evaluate holds at top-1 62.5% / top-3 87.5% over 8 cases, byte-identical to main.

The only golden-report movement is a displayed stem in a reason string:

-  content matches task terms: discount, order, total, code
+  content matches task terms: discount, order, total, cod

Scores (17 / 8 / 6), ordering, and confidence are unchanged. Reasons have always displayed stems rather than the original words (ignor, valu, creat, invoic are already there), so this is consistent with existing output.

Test

Extends the existing round-trip table in signals.test.ts from 5 pairs to 18. Asserting that an inflected form and its base stem to the same set, rather than to a specific literal string, is what keeps the -ed, -ing, -es, -s, and trailing-e rules from disagreeing again — the previous literal-string assertions only ever covered pairs that happened to work, which is how both this regression and the original #69 got through.

npm run ci passes end to end locally, including the regenerated Action bundle.

🤖 Generated with Claude Code