v0.8.42 — Native BGE-M3 sparse channel (R10, #132)

Highlights

• Native BGE-M3 sparse channel (R10, issue #132) — unblocked upstream (fusion-mlx#1621 shipped in PR #1627). POST /v1/embeddings with "return_sparse": true returns BGE-M3 lexical weights per input.
  • EmbeddingClient.embed_batch_sparse() fetches dense + sparse in ONE call (zero extra model passes); per-chunk weights persist to {storage_path}/sparse.db.
  • Hybrid search RRF-fuses a THIRD ranking: sparse dot-product over the candidate pool (vector ∪ BM25).
  • Opt-in FUSION_RAG_SPARSE_CHANNEL=1, default OFF: measured ablation (docs/benchmarks/sparse-ablation.json) shows no lift on the recall_bench corpus (rrf3 recall@10 0.9182 vs rrf2 0.9212 — exact-term queries saturate BM25).
  • Models without a sparse head degrade to BM25 automatically (one-shot 400 / missing-field downgrade); sparse teardown on delete/replace is exact-chunk-id and non-fatal.

Verification

• 585 tests passed, 0 failed, 1 skipped (17 new in tests/test_sparse_channel.py)
• ruff check . clean; docs gate clean
• Real BGE-M3 ablation recorded in docs/benchmarks/sparse-ablation.json

🤖 Generated with Claude Code

v0.8.41 — Permission-aware ingestion (R11)

Highlights

• Per-document ACL (R11, #133) — ingestion captures acl metadata (public/tenant/private + owner tenant + user allow-list) on POST /documents, /ingest, and repo-level via GitLoader(acl=...); search enforces it with no caller opt-out.
• Query-time enforcement + instant revocation — the authoritative acl is resolved from the documents table per query (batched); PATCH acl (revoke = {"visibility": "public"}) propagates to /search + /ask + /similar without reindex.
• Caller identity — resolved request tenant (#61/#68) + new X-Fusion-User header. Absent acl = public: zero change for ACL-less KBs.

Tests: 568 passed, 1 skipped.

🤖 Generated with Claude Code

v0.8.40 — Optional Docling layout-aware PDF ingestion (R9)

Highlights

• Optional Docling PDF backend (R9, #131) — layout-aware ingestion: TableFormer ACCURATE table structure + reading order, per-page export with exact page spans. Opt-in via pip install 'fusion-rag[docling]' + per-KB pdf_backend or FUSION_RAG_PDF_BACKEND. Any failure falls back to legacy PyMuPDF — ingest never fails. Scanned-PDF OCR path (#84) unchanged.
• Bench (docs/benchmarks/docling-ablation.json): on column-major cell PDFs legacy row-intact ratio is 0.00 (no chunk holds item+value together) vs docling 1.00 with BM25 recall@10 1.0. Parse cost ~2.3 s/PDF (CPU-pinned; TableFormer breaks under auto-MPS on Apple Silicon).

Tests: 560 passed, 1 skipped.

🤖 Generated with Claude Code

v0.8.39 — Bench-derived chunking profiles per KB (R8)

Highlights

• Bench-derived chunking profiles per KB (R8, #130) — chunk-size ablation over the labeled corpus (real BGE-M3 + BM25) validates the 512/64 default and adds per-KB chunk_profile selection (markdown/prose 512/64, code 384/48, table 256/32). Explicit chunk_size/chunk_overlap overrides win; unknown profile = 400.
• Fixed: KB-level chunk_overlap was dropped at all 4 ingest Chunker call sites — every KB silently used the 64 default. Now passed through.
• Late-chunking spike: BLOCKED upstream — fusion-mlx /v1/embeddings returns pooled vectors only (dahai80/fusion-mlx#1624).

Bench (docs/benchmarks/chunk-size-ablation.json): 512 @ 12.5% overlap wins recall@10 AND map@10; 768 worse on both; 1024 collapses recall to 0.5394.

Tests: 554 passed, 1 skipped.

🤖 Generated with Claude Code

v0.8.38

PPR graph retrieval layer (R7, issue #129)

• GraphRAG.search_ppr(): personalized PageRank over the extracted entity graph — seeds = entities matching query tokens, undirected edge-weighted diffusion, chunks inherit summed entity PPR mass, returned ranked with ppr_score; degrades to [] on empty graph / no seed match; BFS search() untouched
• Additive-only: fused with the vector path via RRF in the ablation; NOT wired into any route (feature off by default until a real multi-hop corpus validation — explicit issue acceptance)
• Ablation (docs/benchmarks/ppr-ablation.json, real BGE-M3, deterministic entity graph): simple-fact subset PPR 1.0000 vs vector-only 0.9182 / BFS 0.9212; multi-hop two-topic subset PPR 0.5119 vs 0.4702 / 0.4702 — PPR strictly best on both
• HippoRAG-2 regression check (vector+PPR >= vector-only on simple facts): PASS
• Caveat recorded: synthetic corpus has no cross-doc entity chains — numbers gate regressions, not the walk advantage; real multi-hop corpus required before enabling
• 5 unit tests (hand-built graph, no LLM); suite 548 passed, 1 skipped

v0.8.37

Rule-based query complexity routing (R6, issue #128)

• Opt-in router on /ask: FUSION_RAG_ROUTER=auto or per-request route param. Route map: simple -> single-path retrieval (no hybrid/rerank/retry), standard -> hybrid + rerank, complex -> full pipeline + 2 CRAG retry rounds (session history forces complex)
• Deterministic rules only (multi-hop markers, length bounds, question-mark count, history presence) — zero classification cost; learned classifier deferred until rules prove insufficient
• Decision logged per query and echoed as answer["route"]
• Measured A/B on the 55-query labeled corpus (real BGE-M3, docs/benchmarks/router-ablation.json): routed recall@10 0.9182 vs always-full 0.9212 (−0.003, inside the eval-gate tolerance), route mix 49 simple / 6 standard, simple-bucket latency p50 3.4 ms vs 5.0 ms (−32%). Default off.
• 10 unit tests; suite 543 passed, 1 skipped

v0.8.36

Bounded CRAG retry loop (R5, issue #127)

• /ask gains an opt-in grade → rewrite → re-retrieve loop (CRAG / LangGraph agentic-RAG pattern): crag_rounds param or FUSION_RAG_CRAG_ROUNDS env (default 0 = off, single-shot unchanged)
• engine/crag.py: RetrievalGrader (strict LLM judge, structured JSON, verified live that off-topic queries grade 0 relevant) + CorrectiveRAG (bounded rounds, dedupe merge by chunk_id, every grader/rewrite failure degrades to current results with a loud log — enhancement never breaks retrieval)
• Observability: per-round trace as answer["crag"]; CRAG rewrites chain into the citation audit so every retry is reconstructable
• Measured A/B (docs/benchmarks/README.md): retry rate 0.00 on the single-hop bench corpus at min_relevant 1/3/6 — top-10 always grader-sufficient; overhead is one grading call (~46 ms p50 cached, ~0.6 s uncached). Loop target case (multi-hop weak first retrieval) not present on this corpus — re-run there before quoting lift
• Wiring: 6 unit tests; live smoke (crag_rounds=1 → [initial, sufficient] trace); suite 533 passed, 1 skipped

v0.8.35

Contextual BM25 ablation — context prefix lifts BM25 (R4, issue #126)

• Verified the production path already feeds chunk.context into the BM25 index (BM25Index.add_documents indexes context + text)
• New recall_bench.run_contextual_bm25_ablation + scripts/contextual_bm25_ablation.py (PR #147): deterministic labeled corpus, dense channel (BGE-M3) embedded once and shared by both arms; only the BM25 text differs. prefix arm = deterministic template proxy (doc-id + first 240 chars of the parent doc)
• Live result (docs/benchmarks/contextual-bm25-ablation.json): recall@5 0.7939 → 0.8333, recall@10 0.9212 → 1.0, ndcg@10 0.9469 → 1.0, map@10 0.9212 → 1.0 — the production LLM-generated Contextualizer context is strictly richer, so enabling it is non-regressive for the BM25 channel (synthetic-corpus caveat documented)
• Wiring test tests/test_contextual_bm25_ablation.py (mocked embeddings); suite 527 passed, 1 skipped

v0.8.34 — Rerank pool A/B: 20 saturates (R3)

v0.8.34

• Rerank candidate-pool A/B harness (run_rerank_pool_ablation + scripts/rerank_pool_ablation.py)
• Live bge-reranker-v2-m3 result: metrics identical at pool 20/50/100/150, latency 114→1298 ms p50 — rerank_top_n=20 stays default; recorded in docs/benchmarks/rerank-pool-ablation.json

v0.8.33 — PyPI packaging + MCP default KB + venv contract

v0.8.33

• PyPI packaging: pyproject metadata complete, fusion-core>=0.12.1 resolves from PyPI, Trusted Publishing workflow (pypi.yml, OIDC) — pip install fusion-rag after the first tag publishes
• MCP kb_search / kb_ask accept optional kb_id (#136): single-KB default / most-recent fallback, ACL-gated
• start.sh venv discovery contract (#137): VIRTUAL_ENV > project-adjacent .venv > monorepo; loud failure when fusion_rag not importable