test(kb): thorough end-to-end PDF ingest + indexing + search validation#31
Merged
Conversation
Knowledge search is a principal feature; this adds a thorough end-to-end suite that builds a real PDF in-memory (read back by the same pypdf the app uses) and drives the WHOLE pipeline: ingest -> encrypted store -> embeddings index -> every search path. Coverage (18 tests): - PDF ingestion: text extracted from every page (incl. deep pages); non-PDF rejected. - Encryption: document plaintext never on disk. - Lexical search: finds a token on every page (not just the head/title), case-insensitive, isolates the matching doc, ranks by frequency, snippet carries context, absent token -> none. - Semantic search: ranks the meaning-matching doc, skips model/dim mismatches, and — key — a >4000-char doc stores MULTIPLE chunk vectors so a token in a LATE chunk is still found. - Reindex: a doc added while the gateway was down is backfilled and becomes semantic-searchable. - Chat kb_search tool: finds by meaning, and (the recently-fixed bug) finds an UN-INDEXED doc by keyword when a semantic model is available. - HTTP API: upload a PDF then find it via /api/kb/search (lexical + semantic), semantic degrades to lexical when the gateway is down, and reindex recovers it. Embeddings are faked deterministically (an indicator vector over the known test tokens) so semantic ranking + the full index path are exercised without a live embed model. Lexical, ingestion, encryption, and the HTTP layer run against the real code. Mutation-verified: the un-indexed and multi-chunk tests fail if kb_search goes semantic-only or embedding goes head-only.
Factors the in-memory PDF generator into tests/_pdfgen.py (shared) and adds test_knowledge_real_embed.py — an opt-in test that drives the real Knowledge pipeline against a LIVE embeddings endpoint and asserts meaning-based retrieval. Skipped by default; runs only when SMARTBRAIN_TEST_EMBED_URL + _MODEL (+ optional _KEY) are set (and skips gracefully if the endpoint is unreachable). Queries are paraphrases that share no content words with the documents, so only genuine embeddings can answer them — keyword search cannot. Validated live against MLX bge-m3-mlx-fp16 (1024-dim): 5/5 paraphrase queries retrieved the correct passage as top-1 with +0.13..+0.20 score margins over the best decoy. The deterministic CI suite is unchanged (18 pass); full suite green (646, +1 opt-in skip).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Thorough validation that Knowledge indexing + search work — a principal feature that must work flawlessly. Two layers: a deterministic CI suite, and an opt-in test against a real embedding model.
1. Deterministic end-to-end suite (18 tests, runs in CI)
Builds a real PDF in-memory (read back by the same
pypdfthe app uses) and drives the whole pipeline:ingest → encrypted store → embeddings index → every search path.kb_search/api/kb/search(lexical + semantic); degrades to lexical when gateway down; reindex recovers itEmbeddings faked deterministically (indicator vectors) so semantic ranking + index path run without a live model. Mutation-verified: breaking
kb_searchto semantic-only and embedding to head-only makes the relevant tests fail, then pass once restored — the tests have teeth.2. Opt-in real-model test (
test_knowledge_real_embed.py, skipped in CI)Drives the real pipeline against a live OpenAI-compatible embeddings endpoint. Queries are paraphrases sharing no content words with the documents, so only genuine embeddings can answer them (keyword search can't). Runs only when
SMARTBRAIN_TEST_EMBED_URL+_MODEL(+ optional_KEY) are set; skips gracefully if unreachable.Validated live against MLX
bge-m3-mlx-fp16(1024-dim): 5/5 paraphrase queries retrieved the correct passage as top-1, margins +0.13…+0.20 over the best decoy (e.g. "at what time did the ship leave port" → the vessel/harbor passage at 0.834 vs 0.638).Test-only; no production code changes. Full backend suite green (646, +1 opt-in skip), ruff clean.