v4.7.0 — BM25 Fragment Query Support (YANKED — use v4.7.1)

⚠️ YANKED — use v4.7.1

This release contained real deployment code names in tests + CHANGELOG examples. Superseded by v4.7.1 which sanitizes those references. The fix behavior is identical between v4.7.0 and v4.7.1.

• PyPI: v4.7.0 is yanked
• NPM: v4.7.0 is deprecated
• Docker: ghcr.io/lyonzin/knowledge-rag:v4.7.0 remains (tags immutable); latest points to v4.7.1

Please upgrade:

pip install --upgrade knowledge-rag

---

Highlights (unchanged from v4.7.1)

BM25 tokenizer now supports fragment queries. Before v4.7.0, RULE-A002 was indexed as a single token rule-a002, so queries like A002, RULE, B005 silently returned NO_RESULTS. This hit every hyphenated code taxonomy in typical infosec / doc corpora (RULE-*, CVE-*, ADR-*, MS17-*, PROJECT-Custom001-xxxx).

Now emits both the composite AND its sub-parts of length ≥ 2:

bm25.search("A002")   # ← was []; now returns RULE-A002 doc
bm25.search("RULE")   # ← was []; now returns all RULE-* family
bm25.search("B005")   # ← was []; now returns RULE-B005 doc

IDF preserves ranking — composite matches still rank above fragment matches (composite is rarer).

Perf-Safety Heuristic

Sub-token expansion triggers only when the composite contains at least one digit. Alphanumeric codes (RULE-A002, CVE-2024-1234, MS17-010) expand — that's the real use case. Natural-language hyphenated phrases (pass-the-hash, state-of-the-art) stay as single tokens. Without this heuristic, concurrent BM25 query throughput regressed +50-60% on corpora containing common infosec vocabulary.

⚠️ Upgrade Note

Run reindex_documents(force=True) once after upgrade to rebuild the BM25 inverted index with the new sub-token emission. ChromaDB vectors are untouched. Expect ~30-50% increase in BM25 index memory — negligible for typical corpora.

Closes #140 — full technical writeup in PR #141.