Skip to content

GoldPan AI v2.1.0: Reciprocal Rank Fusion, Vietnamese Hybrid BM25 & Idempotent KB

Choose a tag to compare

@ptai-eng ptai-eng released this 25 Oct 04:02
· 11 commits to main since this release

Highlights

GoldPan AI v2.1.0 upgrades the Dual-Mode Knowledge Base with 3-Signal Reciprocal Rank Fusion (RRF), Vietnamese-Aware Hybrid BM25 + Diacritic Folding, Idempotent Content-Hash Deduplication, and explicit 768-dim Matryoshka L2 Normalization for gemini-embedding-001.


✨ What's New in v2.1.0

  • 3-Signal Reciprocal Rank Fusion (RRF, $k=60$)
    • Replaces raw cross-collection distance sorting with scale-invariant Reciprocal Rank Fusion across goldpan_kb_local (384d ONNX), goldpan_kb_gemini (768d Gemini MRL), and a local BM25 lexical ranker.
    • Exposes rrf_score in both API citations and the goldpan_kb_search MCP tool.
  • Vietnamese & Multilingual Offline Retrieval (_tokenize_multilingual + Unicode NFC)
    • Normalizes all extracted Markdown and search queries to composed Unicode (NFC) to fix decomposed (NFD) Vietnamese diacritics from PDFs and Word files.
    • Combines dense ONNX embeddings with an offline BM25 ranker supporting accented unigrams, ASCII-folded unigrams (đ/Đ -> d), and 2-syllable compound bigrams (tinh_diem, ban_to_chuc), enabling accurate offline retrieval for both accented and unaccented Vietnamese queries.
  • Idempotent Content-Hash Deduplication (compute_document_id)
    • Generates deterministic uuid5 document IDs from normalized content and source metadata so re-ingesting a document via the UI, CLI, or MCP updates vectors and KB/*.md files in place without creating duplicates.
  • Explicit 768d Matryoshka Gemini Embeddings (outputDimensionality=768)
    • Explicitly requests 768-dimensional slices from gemini-embedding-001 and applies unit L2 normalization (_l2_normalize) prior to ChromaDB upsert.
  • Expanded Verification Suite (12/12 Passed)
    • Added automated unit and integration tests for 768d MRL payloads, L2 norm verification, HTTP 429 retry, cross-collection RRF fusion, idempotent ingestion, and Vietnamese NFC/NFD + unaccented retrieval (run_tests.py & backend/tests/test_api.py), plus clean tsc --noEmit verification on the Next.js frontend.