Skip to content

v0.16.1

Choose a tag to compare

View all tags
@github-actions github-actions released this 18 Apr 13:16
v0.16.1
1a7b03e

What's new

Tightens CJK keyword search to eliminate false positives while keeping legitimate OCR-spaced matches. Builds on the CJK segmentation shipped in v0.16.0.

What was wrong

NEAR(c1 c2 c3, n-1) on the per-character index can match documents where the query's characters appear close together but not in the queried phrase. For example, searching "能智工人" against a document containing "人工智能是计算机科学的重要分支" returned a hit even though the exact phrase is not present — all four characters happen to be within a 3-token window. Previously, any FTS5 NEAR candidate that failed phrase/term/title verification fell back to a reference-only block-0 row, so these false positives still surfaced in results.

Separately, PDFs whose OCR inserts spaces between CJK characters (e.g. 人 工 智 能 是 ...) normalized to text that could not be matched against a connected query like 人工智能, so legitimate hits were missed.

The fix

  • buildKeywordSearchRow now returns null when no exact phrase, term, or title evidence backs an FTS5 candidate, instead of falling back to block 0. Unverified hits no longer leak into results.
  • normalizeExactText collapses consecutive single-CJK-character tokens back into a word, so "人 工 智 能" normalizes to "人工智能" and OCR-spaced content stays matchable against a connected query. Multi-character CJK tokens are left alone, so genuine word boundaries like "机器学习 人工智能" are preserved.

No migration needed — the index format is unchanged from v0.16.0.

Full Changelog: v0.16.0...v0.16.1

Assets 4
Loading