Repository navigation
Minor release. Nothing breaking on upgrade, and existing indexes need no reindex.
Most of it answers six reports from one large installation (a whole drive as documents_dir, about 610,000 chunks). Thank you, @MagnetoPause.
Highlights
- File watcher: it applies
exclude_patternsbefore scheduling anything, so files written inside excluded folders (TEMP, tool output, venvs) no longer start reindex passes (#295). It reindexes only the paths that changed, through the newindex_paths(); before, any event walked and checked every indexed document (#297). The first pass after a start, a pass with more than 500 changed paths and a pass with a repair pending are still full scans. An error while handling one event no longer stops the watcher. - Scans: an exclude pattern of the form
folder/*prunes the folder during the walk instead of visiting every file in it. Which files are excluded does not change (#296). - Search on large indexes: a document write or a server restart no longer makes the next search wait minutes for BM25 (#299). Above 20,000 chunks the rebuild runs on a background thread while searches use the published generation — or, right after a start, the semantic channel alone. A keyword-only search (
hybrid_alpha=0) waits for the first build. Up to 20,000 chunks nothing changes: the search right after a write still sees it. - Result diversity: MMR now runs when the reranker is enabled (#294). A chunk with the same words as a result already selected is returned only when no distinct candidate is left; near-duplicates are weighed against relevance. Results stay ordered by score.
- Index hygiene: a reindex removes orphan chunks — chunks whose document dropped out of the catalogue — and reports them as
orphan_chunks_removed. A file that is still in place and only lost its catalogue entry is registered again, not removed (#298). - Evaluation:
evaluate_retrievalsearches withhybrid_alpha=0.3, the default ofsearch_knowledge. It used 0.5, so its figures did not describe what MCP clients get by default. Figures recorded with an earlier version are not comparable: record a new baseline. - Reindex diagnostics:
get_reindex_statuslists the files that failed the last reindex, with the reason (#288). - Recall probe:
get_index_statsreportsself_recall, andsearch.ef_searchsets the HNSW search depth for collections created from now on (#287).
Known limits
- On an index above 20,000 chunks, a keyword match for content written just before can be missing until the background refresh is published. Semantic matches are not affected.
- A BM25 rebuild still reads every chunk and holds the index lock while it runs; a write that arrives meanwhile waits for it. Updating BM25 incrementally is not part of this release.
What's Changed
- fix(reindex): report the files that failed in get_reindex_status (#288) by @lyonzin in #289
- fix(search): make HNSW ef_search configurable and report self-recall (#287) by @lyonzin in #290
- docs(readme): center the badges block by @lyonzin in #291
- docs(readme): render badges as HTML inside the centered block by @lyonzin in #292
- docs(readme): close the badges div so the stats list stays left-aligned by @lyonzin in #293
- fix(search): run MMR deduplication when the reranker is enabled by @lyonzin in #302
- fix(watcher): apply exclude_patterns before scheduling a reindex by @lyonzin in #303
- fix(ingestion): prune excluded folders during the scan walk by @lyonzin in #304
- fix(watcher): reindex only the changed paths instead of a full scan by @lyonzin in #305
- fix(search): refresh stale BM25 in the background instead of blocking queries by @lyonzin in #306
- fix: keep the exclude, watcher and BM25 fixes safe on edge cases by @lyonzin in #308
- fix(reindex): remove chunks of documents the catalogue no longer knows by @lyonzin in #307
- fix(tools): measure evaluate_retrieval with the default alpha of search_knowledge by @lyonzin in #309
- chore(release): v5.1.0 by @lyonzin in #310
Full Changelog: v5.0.1...v5.1.0