Skip to content

v0.3.0

Choose a tag to compare

@odiak odiak released this 27 Sep 00:52
· 19 commits to main since this release
fd1e98a

The built-in full-text index (R2) is rebuilt as incremental segments, so it keeps working on large vaults. No host-side code changes are needed; existing vaults migrate automatically on their first access after deployment.

  • The index is now a set of immutable segments. Each indexing pass writes only the notes that changed into one segment (at most 2 million code units per pass, a few seconds), so the cost of an update is proportional to the change, not to the vault. The initial indexing is split into passes the same way and stays under the Durable Object CPU and memory limits (measured: a 100 MB vault of 12,720 notes indexes in about 2 minutes).
  • Idle passes merge small segments automatically (compaction, without re-tokenizing).
  • Ranking is now BM25 over the query phrases (it used to be matches / log2(length), with no IDF).
  • The previous whole-vault generation is retired and deleted automatically once every note is in a new segment.
  • indexStatus().fts.pending (notes not yet in the built-in index) is added.
  • VaultBindings.ftsMaxTotalCodeUnits now bounds the whole index (default 50,000,000 code units ≈ 120 MB of Japanese Markdown; in 0.2.0 it bounded one rebuild at 8,000,000). Exceeding it is still reported in indexStatus().fts.error.
  • A rebuild that keeps dying mid-way is recorded in indexStatus().fts.error instead of being retried forever (up to 0.2.0 it retried every 60 seconds).
  • Omitting VaultBindings.vectorize / embedder runs without semantic search (no embeddings, no Vectorize). In this repository's Worker that is the SEMANTIC_SEARCH=off variable.

R2 layout: fts/{tenant}/{db}/manifest.json plus seg-*/. A manifest.json written by 0.2.0 or earlier (version 1) is still read.