Repository navigation
GoldPan AI v2.1.0: Reciprocal Rank Fusion, Vietnamese Hybrid BM25 & Idempotent KB
Highlights
GoldPan AI v2.1.0 upgrades the Dual-Mode Knowledge Base with 3-Signal Reciprocal Rank Fusion (RRF), Vietnamese-Aware Hybrid BM25 + Diacritic Folding, Idempotent Content-Hash Deduplication, and explicit 768-dim Matryoshka L2 Normalization for gemini-embedding-001.
✨ What's New in v2.1.0
-
3-Signal Reciprocal Rank Fusion (
RRF,$k=60$ )- Replaces raw cross-collection distance sorting with scale-invariant Reciprocal Rank Fusion across
goldpan_kb_local(384d ONNX),goldpan_kb_gemini(768d Gemini MRL), and a local BM25 lexical ranker. - Exposes
rrf_scorein both API citations and thegoldpan_kb_searchMCP tool.
- Replaces raw cross-collection distance sorting with scale-invariant Reciprocal Rank Fusion across
-
Vietnamese & Multilingual Offline Retrieval (
_tokenize_multilingual+ UnicodeNFC)- Normalizes all extracted Markdown and search queries to composed Unicode (
NFC) to fix decomposed (NFD) Vietnamese diacritics from PDFs and Word files. - Combines dense ONNX embeddings with an offline BM25 ranker supporting accented unigrams, ASCII-folded unigrams (
đ/Đ -> d), and 2-syllable compound bigrams (tinh_diem,ban_to_chuc), enabling accurate offline retrieval for both accented and unaccented Vietnamese queries.
- Normalizes all extracted Markdown and search queries to composed Unicode (
-
Idempotent Content-Hash Deduplication (
compute_document_id)- Generates deterministic
uuid5document IDs from normalized content and source metadata so re-ingesting a document via the UI, CLI, or MCP updates vectors andKB/*.mdfiles in place without creating duplicates.
- Generates deterministic
-
Explicit 768d Matryoshka Gemini Embeddings (
outputDimensionality=768)- Explicitly requests 768-dimensional slices from
gemini-embedding-001and applies unit L2 normalization (_l2_normalize) prior to ChromaDB upsert.
- Explicitly requests 768-dimensional slices from
-
Expanded Verification Suite (
12/12 Passed)- Added automated unit and integration tests for 768d MRL payloads, L2 norm verification, HTTP 429 retry, cross-collection RRF fusion, idempotent ingestion, and Vietnamese NFC/NFD + unaccented retrieval (
run_tests.py&backend/tests/test_api.py), plus cleantsc --noEmitverification on the Next.js frontend.
- Added automated unit and integration tests for 768d MRL payloads, L2 norm verification, HTTP 429 retry, cross-collection RRF fusion, idempotent ingestion, and Vietnamese NFC/NFD + unaccented retrieval (