bug: -ed/-es stemming strips one character, so task words never match their base identifier

[aryamthecodebreaker]

Summary

normalizeToken() in packages/core/src/signals.ts strips only one character for the -ed and -es suffixes, so a task word and the identifier it refers to stem to different tokens unless the base form happens to end in e.

if (token.length > 5 && token.endsWith("ies")) return `${token.slice(0, -3)}y`;
if (token.length > 5 && token.endsWith("ing")) return token.slice(0, -3);
if (token.length > 4 && token.endsWith("ed"))  return token.slice(0, -1);  // strips "d" only
if (token.length > 4 && token.endsWith("es"))  return token.slice(0, -1);  // strips "s" only
if (token.length > 3 && token.endsWith("s"))   return token.slice(0, -1);

Reproduction

node -e "const {tokenizeText}=require('./packages/core/dist/signals.js'); \
for (const [a,b] of [['failed','fail'],['boxes','box'],['routing','route'],['parsing','parse']]) \
  console.log(a, '->', [...tokenizeText(a)], '|', b, '->', [...tokenizeText(b)]);"

failed  -> [ 'faile' ]  | fail  -> [ 'fail' ]     mismatch
boxes   -> [ 'boxe'  ]  | box   -> [ 'box'  ]     mismatch
routing -> [ 'rout'  ]  | route -> [ 'route' ]    mismatch
parsing -> [ 'pars'  ]  | parse -> [ 'parse' ]    mismatch
created -> [ 'create']  | create-> [ 'create' ]   match (base ends in "e")

Impact

This is the matching function behind path overlap, content overlap, and findRelatedTests() — the heart of the ranker. A very common phrasing like "password reset failed" does not match a fail/failure identifier or a retry-failed.ts path. Two separate rules interact badly:

• -ed / -es under-strip (failed -> faile).
• -ing over-strips relative to an -e base (routing -> rout, but route -> route).

So the two halves of the stemmer disagree about whether a trailing e survives, and neither direction reliably converges on the base form.

Suggested fix

Make the suffix rules converge on one normal form. A minimal version that fixes all four cases above: after stripping -ed / -ing / -es, also strip a trailing e (or equivalently, normalize both route and routing to rout). Anything that makes stem(x) === stem(inflect(x)) hold is fine — the important part is that the rules agree.

Because this changes ranking, it should be landed together with npm run evaluate results and, ideally, a couple of new benchmarks/cases.json entries using past-tense task phrasing.

Note: apps/web/app/demo.tsx contains a hand-copied version of the same rules and has the same defect — see the separate issue on demo/core tokenizer drift.

[aryamthecodebreaker]

Follow-up while reviewing main: the existing test is why this went unnoticed.

packages/core/test/signals.test.ts covers stemming with exactly two inflected words:

it("normalizes simple plural and verb forms", () => {
  const signals = extractTaskSignals({ issueText: "Invoices are created for users" });
  expect(signals.tokens.has("invoice")).toBe(true);
  expect(signals.tokens.has("create")).toBe(true);
  expect(signals.tokens.has("user")).toBe(true);
});

invoices -> invoice and created -> create are both cases where stripping a single character is correct, because the base form already ends in e. The test asserts the half of the rule that works and never touches the half that does not (failed, boxes, routing, parsing).

Any fix should extend this test with those cases, ideally as a round-trip assertion — stem(inflected) === stem(base) — so the -ed/-es and -ing rules cannot disagree about a trailing e again.