v4.10.0 - Indexing reliability, bounded memory and scoped retrieval

knowledge-rag 4.10.0 improves indexing recovery, bounds retained memory and adds folder-scoped retrieval. It incorporates the fixes proposed for the unpublished 4.9.3 release.

Changes

• Search selected folders: optional include_folders and exclude_folders apply recursively before top-k selection in Chroma, BM25 and FTS5. Category filters still intersect with the selected scope. Absolute paths, traversal and escaping links are rejected (#232).
• Keep useful candidates from both retrieval channels: semantic and keyword candidates share the existing bounded reranker budget without letting the public result cap prematurely truncate the semantic pool. The default BGE model, hybrid weight and explicit single-channel modes are preserved.
• Recover interrupted mutations: restore the document inventory before vector rollback and persist a bounded repair journal when rollback itself fails. Incremental indexing repairs affected sources after restart, including incomplete-add orphans. Backup expiration requires evidence of a committed replacement collection.
• Reduce retained memory and repeated work: aggregate metric observations, pool BM25 vocabulary, avoid retaining historical mutation payloads, skip unchanged files before parsing, bound pending indexing work and index FTS5 chunk identities for updates.
• Fix Windows startup and storage edge cases: resolve relative preflight roots (#233), use non-signaling process checks and OS file locks, preserve long-path identities, and release native PDF resources after failed construction.
• Choose batch sizes from the actual execution provider: CPU defaults to 32 (#224), CUDA to 256 and explicit Windows DirectML to 8. DirectML requires an adapter index and uses serialized inference. Automatic selection remains CUDA/CPU.
• Correct operational documentation and packaging: multi-client HTTP setup (#223), recovery procedures, model migration, GPU installation, bundled presets and installer behavior. Dependency updates and the NPM OIDC publication workflow are included.

Validation

The final Windows runtime passed 1,263 tests, with 1 skipped and 83.57% coverage; all 5 separately selected chaos tests passed. The independent reviewer verified all six reproduced findings, including 16/16 recovery scenarios, 8/8 recoveries in fresh Python processes and 62 additional focused tests. Local pre-commit hooks, strict mypy selection, API compatibility, version consistency and security checks passed.

All 35 PR checks passed on the final head, including the nine Linux/macOS/Windows and Python 3.11/3.12/3.13 combinations: CI, Quality Gate.

The final real Windows CUDA reindex attempted 5,962 files and indexed 5,951 sources / 70,083 chunks. Eleven empty or invalid sources remained explicit errors. Chroma, FTS5 and BM25 counts agreed; the 384-dimensional vectors were finite and nonzero, and the source/configuration hashes remained unchanged. The indexing call took 476.009 seconds; the complete monitoring window, including preparation and validation, was 533.403 seconds. Peak process RSS was 2,473,619,456 bytes. This is one local workload, not a general throughput guarantee.

Development retrieval diagnostics on 60 positive questions improved expected-source hits@10 from 33 to 37 and evidence hits@10 from 28 to 31. These questions were used during development and are not an unseen quality estimate. An isolated multilingual E5 comparison did not justify changing the default model. A source match does not establish the correctness or completeness of a generated answer.

The separate final reserve contained six PT/EN pairs, twelve questions, frozen before evaluation and never used for tuning. Against the pre-review audit candidate, expected-source hits@10 improved 4/12 to 6/12, evidence hits@10 3/12 to 4/12, and MRR@10 0.2619 to 0.3361. Hits@1 stayed at 3/12. All gains came from English questions: Portuguese source/evidence hits remained 0/6 in both versions. This small paired sample exposes a remaining cross-language retrieval limitation; it does not establish general multilingual quality.

Across the full evaluation, 180 direct calls and 180 MCP calls agreed, including both auto and hybrid methods. Seventeen folder-filter scenarios passed through both interfaces. The six negative controls returned approximate context but no matching fictitious marker; no answer generator or abstention benchmark was evaluated.

CPU, CUDA and AMD DirectML (Windows, Radeon integrated adapter 1) passed the same seven-source / 162-chunk real-storage scenario through three rounds, including edits, deletion, forced indexing, reopen and concurrent queries. The CPU stress case for #224 indexed 331 JS/JSON chunks, with 330 sequences reaching 512 tokens and observed ONNX batches capped at 32. All 384-dimensional vectors were finite and no OOM occurred; the indexing call took 18.5486 seconds with 1,477,693,440 bytes peak RSS. This validates the tested workload and hardware, not every host memory budget or GPU model.

Workload-specific memory diagnostics reduced retained Python BM25 allocations from 120,711,243 to 26,859,694 bytes for 10,000 documents. For 100 updates within 100,000 FTS5 rows, elapsed time fell from 1.754 to 0.0618 seconds; initial insertion increased from 0.412 to 0.800 seconds. These measurements describe those workloads rather than universal throughput guarantees.

The final performance gate compared 10 compatible benchmarks at the unchanged 10% threshold, with no excessive regression. Cache-hit and cache-miss benchmark latency decreased by 43.6% and 69.5%, respectively. Thirteen rows with changed workloads or missing historical RSS byte measurements were explicitly marked incomparable, rather than counted as improvements.

Upgrade notes

• Stop legacy server instances before upgrading to the OS file-lock protocol. Do not delete a live .guard sidecar.
• No required index-format migration is introduced. Keep different embedding models in separate data and metadata paths.
• Windows DirectML remains explicit and experimental. Validation on one AMD Windows adapter does not establish Linux ROCm or Apple GPU support. See the GPU guide.
• Repair records and preserved backups improve recovery from handled failures; they are not a power-loss atomic transaction guarantee. Read the recovery procedures.

Integrated through #234, including contributions from #222, #225, #226 and #227–231. Closes #223, #224, #232 and #233.

Details: changelog, independent review and retrieval audit, search quality and folder scopes.