Skip to content

v0.8.0

Choose a tag to compare

@bluesky585 bluesky585 released this 18 Sep 03:20
· 12 commits to main since this release
c4758f4

Two retrieval-quality fixes since v0.7.0. No API breaks; drop-in upgrade.

BM25 now works for Chinese, Japanese, and Korean (#63)

Sparse search previously split a run of Han/kana/hangul whole, so a
paraphrased CJK query scored zero on every document. CJK text is now
segmented into sliding bigrams — the unit Lucene's CJK analyzers use,
no dictionary needed — while Latin words pass through unchanged.
Bigrams never cross a script boundary.

查询「如何提高检索的效果」此前在含「提高检索质量」的文档上得 0 分,现在正常命中。

Over-budget sentences are re-cut at their clauses (#65)

sentence and semantic sent a sentence that exceeded the size budget
straight to token windows, cutting words wherever the window boundary
landed. The sentence now gets a second, finer split first — commas,
then whitespace, then tokens — via the new shared recursive.FallbackRules().
Text with no finer delimiter behaves exactly as before.

Also

  • examples/ — six runnable programs, one per idea (#64)
  • Fuzz targets for the new paths: 90s BPE + 60s standard, no findings
  • Verified against three real scraped corpora (English docs, Chinese article, Markdown syntax)

Full changelog: v0.7.0...v0.8.0