Skip to content

v0.5.0

Choose a tag to compare

@odiak odiak released this 28 Sep 03:36
· 14 commits to main since this release
64ac31c

The built-in full-text index no longer breaks on very large notes or on vaults that are edited a lot. No host-side code changes are needed; existing indexes keep working as they are.

  • Large notes no longer take the index down. A single note of 1–2 million characters used to push the Durable Object past its 128 MB memory limit, and after three resets the index stopped updating for the whole vault. Tokenization now streams straight into the postings (a 2M-character note peaks at ~65 MB instead of ~265 MB, same CPU), and a note is indexed up to its first million characters rather than being skipped entirely.
  • Random-looking text is bounded. A segment holds at most 400k distinct terms. A note that would overflow a segment starts the next one; only a note that overflows on its own (e.g. mojibake, where every character bigram is distinct) is left out.
  • Replaced versions are shed. Old versions of edited notes stayed in their segments until a merge happened to include them, and they counted toward the size guard, so a vault could reach the guard by being edited rather than by growing. Idle passes now rewrite a segment once a quarter of its text (and at least 250k characters) is dead weight, and a pass that hits the guard sheds such text first, stopping only when nothing can be freed.
  • ftsRebuild can recover a vault that hit the guard. The rebuild records an epoch; segments from before it are ignored by the guard and retired in one step once every note has been re-indexed. Searches keep working throughout.
  • Over-long queries are refused instead of failing. grepNotes and VaultClient.grep() return { status: "query-too-long", maxTokens } for queries of more than 32 index terms (about 60 CJK characters), which would otherwise exceed the Worker's subrequest limit on a large index.

The README now has a "Search coverage and limits" section (CJK support, tested vault size, the per-note cut-off, query length).

Library API: FullTextSearchResult gains the query-too-long variant; appendFtsSegment() returns { manifest, segment, docs, dropped } (with manifest/segment null when nothing was written) instead of null; buildIndex() takes maxTerms and reports dropped.