bug: web demo reimplements the core tokenizer and has drifted from it

[aryamthecodebreaker]

Summary

apps/web/app/demo.tsx hand-copies the core tokenizer, and the copy has already drifted from packages/core/src/signals.ts. The site claims the demo mirrors the CLI:

"This browser demo ranks a safe sample repository. The CLI applies the same transparent ideas to your real checkout." — apps/web/app/page.tsx

It does not.

Differences

demo.tsx:

function tokens(value: string) {
  return new Set(value.toLowerCase().split(/[^a-z0-9]+/).map((token) => {
    if (token.length > 5 && token.endsWith("ies")) return `${token.slice(0, -3)}y`;
    if (token.length > 4 && token.endsWith("ed"))  return token.slice(0, -1);
    if (token.length > 4 && token.endsWith("es"))  return token.slice(0, -1);
    if (token.length > 3 && token.endsWith("s"))   return token.slice(0, -1);
    return token;
  }).filter((token) => token.length > 2));
}

versus tokenizeText / normalizeToken in core:

Behaviour	core	demo
-ing rule	yes	missing
stop-word list (~130 words)	yes	missing — the, this, from all score
camelCase split (resetToken -> reset token)	yes	missing
stop-word filter re-applied after stemming	yes	missing

The scoring formulas diverge too: core weights path overlap 3 / content overlap 2 capped at 8, plus changed-file, mention, kind, deployment, and import-proximity terms; the demo is path*3 + content*2 + (code ? 2 : 0). Confidence thresholds differ as well (core: changed-or->=14 high, >=8 medium; demo: >=10 / >=6).

Impact

The demo is the top-of-funnel artifact for a tool whose entire pitch is transparent, inspectable ranking. A visitor who tries a phrase in the browser and then runs the CLI can get a different answer, which undermines exactly the claim the page is making. The duplication also means the -ed/-es stemmer bug filed separately has to be fixed in two places, and nothing fails if only one is fixed.

Suggested fix

Stop maintaining a second implementation. tokenizeText is pure, dependency-free, and already exported from @aryam/fixmap-core, so the demo can import the real function and run it in the browser — the sample repository stays hardcoded, only the ranking logic is shared.

If bundling core into the web app is undesirable, the fallback is to derive the demo from core at build time and add a test asserting the two produce identical output on a fixed corpus, so drift fails CI instead of shipping.

[aryamthecodebreaker]

Correction to the suggested fix above: tokenizeText is not currently exported from the package barrel, so "import the real function" needs one extra step.

packages/core/src/index.ts exports buildFixMapReport, buildImportGraph, findImportProximity, rankContextFiles, buildRiskNotes, buildSummary, buildTestRoutes, renderJsonReport, renderMarkdownReport, scanRepo, findGatedTestDiagnostics, and the types — but nothing from ./signals.js.

So sharing the tokenizer with the web demo means first adding:

export { extractTaskSignals, extractFileMentions, tokenizePath, tokenizeText } from "./signals.js";

That is arguably worth doing on its own merits — tokenizeText is the function that decides what "matches", and it is the piece a third party would most want when explaining or extending FixMap's ranking. It is pure and dependency-free, so exporting it costs nothing and makes the "transparent, inspectable ranking" claim actually checkable from outside the package.