fix(fieldNorm): count word-starts instead of space transitions to avoid over-counting leading/trailing spaces#830
Merged
Conversation
Leading and trailing spaces were silently inflating numTokens by one boundary each, producing a lower norm value (0.707 instead of 1.0 for a single-word field) and therefore a higher (worse) final search score for any document whose field value contained stray whitespace. Root cause: the original loop started numTokens at 1 and incremented on every space→non-space transition. A leading space was treated as a transition from an implicit word, so ' hello' gave numTokens=2 instead of 1. Similarly, 'hello ' transitioned word→space near the end, giving numTokens=2 even though only one word is present. Fix: rewrite the counter to tally word-starts (non-space character that follows a space or the string start). This is equivalent for typical strings — 'hello world' still yields 2 — but correctly ignores leading and trailing whitespace. Empty and all-whitespace strings are clamped to 1 to preserve the existing fallback behaviour. Test: add a failing test that demonstrates the bug for ' hello', 'hello ', ' hello world ', and ' ', then verify it passes after fix.
krisk
approved these changes
Jun 22, 2026
krisk
left a comment
krisk
left a comment
Owner
There was a problem hiding this comment.
Nice catch, this one's real. " hello" was counting as 2 tokens since values reach norm.get() untrimmed and only all-whitespace gets filtered. Fix looks good and trimmed strings come out the same as before.
Tiny thing: the table says old count for " hello world " is 3, it's actually 4.
Approving, thanks!
krisk
added a commit
that referenced
this pull request
Jun 22, 2026
krisk
pushed a commit
to chatman-media/Fuse
that referenced
this pull request
Jul 3, 2026
The word counter added in krisk#830 only checked for charCode 32 (plain ASCII space), so any field that separated words with a tab or newline instead was scored as a single long word. That gives it an artificially better field-length norm than a same-content field using regular spaces - e.g. 'foo\tbar\tbaz' scores 20x better than 'foo bar baz' even though they're the same three words. Added a small isWordSeparator helper covering the usual whitespace range (tab/LF/VT/FF/CR, space, NBSP) instead of the single charCode check, plus a test that fails without the fix.
krisk
pushed a commit
that referenced
this pull request
Jul 3, 2026
The word counter added in #830 only checked for charCode 32 (plain ASCII space), so any field that separated words with a tab or newline instead was scored as a single long word. That gives it an artificially better field-length norm than a same-content field using regular spaces - e.g. 'foo\tbar\tbaz' scores 20x better than 'foo bar baz' even though they're the same three words. Added a small isWordSeparator helper covering the usual whitespace range (tab/LF/VT/FF/CR, space, NBSP) instead of the single charCode check, plus a test that fails without the fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
fieldNorm's word-counting loop startednumTokensat1and incremented on every space-to-non-space transition. This means a leading space was treated as a transition from an implicit word, over-counting the number of words:numTokensnumTokens" hello""hello "" hello world "" ""hello world"""The inflated count produces a lower norm value (e.g.
0.707instead of1.0for a single-word field), which in turn raises the final multiplicative score — penalising any document whose stored field value happens to contain leading or trailing whitespace.Fix
Replace the space-transition counter with a word-start counter: increment only when encountering a non-space character that follows a space or the start of the string. This is equivalent to the old logic for well-trimmed strings (same result for
"hello world","hello world", etc.) and correctly ignores stray boundary whitespace. Strings with zero real words (empty or all-whitespace) are clamped tonumTokens = 1to preserve the existing fallback.Tests
A new test — "does not count leading or trailing spaces as word boundaries" — was failing before the fix and passes after. All 371 existing tests continue to pass.