Repository navigation
v0.8.0
Two retrieval-quality fixes since v0.7.0. No API breaks; drop-in upgrade.
BM25 now works for Chinese, Japanese, and Korean (#63)
Sparse search previously split a run of Han/kana/hangul whole, so a
paraphrased CJK query scored zero on every document. CJK text is now
segmented into sliding bigrams — the unit Lucene's CJK analyzers use,
no dictionary needed — while Latin words pass through unchanged.
Bigrams never cross a script boundary.
查询「如何提高检索的效果」此前在含「提高检索质量」的文档上得 0 分,现在正常命中。
Over-budget sentences are re-cut at their clauses (#65)
sentence and semantic sent a sentence that exceeded the size budget
straight to token windows, cutting words wherever the window boundary
landed. The sentence now gets a second, finer split first — commas,
then whitespace, then tokens — via the new shared recursive.FallbackRules().
Text with no finer delimiter behaves exactly as before.
Also
examples/— six runnable programs, one per idea (#64)- Fuzz targets for the new paths: 90s BPE + 60s standard, no findings
- Verified against three real scraped corpora (English docs, Chinese article, Markdown syntax)
Full changelog: v0.7.0...v0.8.0