Skip to content

v5.1.0 — Path-level watcher, non-blocking BM25 and orphan cleanup

Latest

Choose a tag to compare

@lyonzin lyonzin released this 14 Nov 16:36
· 2 commits to master since this release
a72441c

Minor release. Nothing breaking on upgrade, and existing indexes need no reindex.

Most of it answers six reports from one large installation (a whole drive as documents_dir, about 610,000 chunks). Thank you, @MagnetoPause.

Highlights

  • File watcher: it applies exclude_patterns before scheduling anything, so files written inside excluded folders (TEMP, tool output, venvs) no longer start reindex passes (#295). It reindexes only the paths that changed, through the new index_paths(); before, any event walked and checked every indexed document (#297). The first pass after a start, a pass with more than 500 changed paths and a pass with a repair pending are still full scans. An error while handling one event no longer stops the watcher.
  • Scans: an exclude pattern of the form folder/* prunes the folder during the walk instead of visiting every file in it. Which files are excluded does not change (#296).
  • Search on large indexes: a document write or a server restart no longer makes the next search wait minutes for BM25 (#299). Above 20,000 chunks the rebuild runs on a background thread while searches use the published generation — or, right after a start, the semantic channel alone. A keyword-only search (hybrid_alpha=0) waits for the first build. Up to 20,000 chunks nothing changes: the search right after a write still sees it.
  • Result diversity: MMR now runs when the reranker is enabled (#294). A chunk with the same words as a result already selected is returned only when no distinct candidate is left; near-duplicates are weighed against relevance. Results stay ordered by score.
  • Index hygiene: a reindex removes orphan chunks — chunks whose document dropped out of the catalogue — and reports them as orphan_chunks_removed. A file that is still in place and only lost its catalogue entry is registered again, not removed (#298).
  • Evaluation: evaluate_retrieval searches with hybrid_alpha=0.3, the default of search_knowledge. It used 0.5, so its figures did not describe what MCP clients get by default. Figures recorded with an earlier version are not comparable: record a new baseline.
  • Reindex diagnostics: get_reindex_status lists the files that failed the last reindex, with the reason (#288).
  • Recall probe: get_index_stats reports self_recall, and search.ef_search sets the HNSW search depth for collections created from now on (#287).

Known limits

  • On an index above 20,000 chunks, a keyword match for content written just before can be missing until the background refresh is published. Semantic matches are not affected.
  • A BM25 rebuild still reads every chunk and holds the index lock while it runs; a write that arrives meanwhile waits for it. Updating BM25 incrementally is not part of this release.

What's Changed

  • fix(reindex): report the files that failed in get_reindex_status (#288) by @lyonzin in #289
  • fix(search): make HNSW ef_search configurable and report self-recall (#287) by @lyonzin in #290
  • docs(readme): center the badges block by @lyonzin in #291
  • docs(readme): render badges as HTML inside the centered block by @lyonzin in #292
  • docs(readme): close the badges div so the stats list stays left-aligned by @lyonzin in #293
  • fix(search): run MMR deduplication when the reranker is enabled by @lyonzin in #302
  • fix(watcher): apply exclude_patterns before scheduling a reindex by @lyonzin in #303
  • fix(ingestion): prune excluded folders during the scan walk by @lyonzin in #304
  • fix(watcher): reindex only the changed paths instead of a full scan by @lyonzin in #305
  • fix(search): refresh stale BM25 in the background instead of blocking queries by @lyonzin in #306
  • fix: keep the exclude, watcher and BM25 fixes safe on edge cases by @lyonzin in #308
  • fix(reindex): remove chunks of documents the catalogue no longer knows by @lyonzin in #307
  • fix(tools): measure evaluate_retrieval with the default alpha of search_knowledge by @lyonzin in #309
  • chore(release): v5.1.0 by @lyonzin in #310

Full Changelog: v5.0.1...v5.1.0