Skip to content

v0.4.0

Choose a tag to compare

@odiak odiak released this 27 Sep 10:07
· 17 commits to main since this release
1cf27f9

Full-text search reads only the postings it needs, which makes searches fast on large vaults. No host-side code changes are needed; existing indexes are rewritten to the new layout automatically after deployment.

  • Each shard is split into 64 buckets, compressed separately, and a search reads only the buckets its query terms hash into, with R2 range reads. The bytes read depend on the number of terms, not on the size of the index (measured on a 100 MB vault of 12,720 notes: long phrases and 4-word AND queries went from 5–7 s to about 1 s, and to 0.2–0.6 s once cached).
  • A phrase query reads the smallest set of bigrams that covers every character (the set of matching notes is unchanged).
  • Segments written by 0.3.0 are rewritten to the new layout one at a time by the maintenance pass (10–20 s each) and remain searchable meanwhile. Upgrading directly from a version-1 index (0.2.0 or earlier) also works.
  • If a rebuild pass was scheduled but its alarm was lost, the next request picks it up again.

R2 layout: seg-*/shard-NNN.bin (concatenated buckets) and seg-*/index.bin (bucket byte ranges) are added; the shard-NNN.bin.gz files of 0.3.0 are still read.