Repository navigation
GoldPan AI v3.0.0 — Multilingual E5 ONNX RAG, Agent Context Firewall & Dual-Track Benchmarks
🚀 What's New in GoldPan AI v3.0.0
GoldPan AI v3.0.0 marks a major architectural milestone, evolving from a multimodal converter into a hardened, privacy-first data refinery, Multilingual E5 ONNX + SQLite FTS5 Hybrid RAG engine, and Agent Context Firewall for autonomous LLM agents (MCP, REST API, CLI).
🏛️ 1. Pillar B — Multilingual & Vietnamese First-Class Core
intfloat/multilingual-e5-smallONNX Embeddings (384-dim): Automaticpassage:prefix on document ingestion,query:prefix on search, and L2 normalization in isolated collectiongoldpan_kb_e5_v3.- Persistent SQLite FTS5 Lexical Index (
chroma_data/goldpan_fts.db): Replaces RAM-heavy in-memory BM25 with persistent disk-backed FTS5 (WALmode) indexing both accentedtext_nfcand ASCII-foldedtext_folded(đ/Đ -> d+ compound bigrams) fused via Reciprocal Rank Fusion (RRF, k=60). - Legacy Vietnamese Encoding Decoder (
vn_encoding.py): Automatically detects and converts VNI-Windows, TCVN3 (ABC), and decomposed Unicode NFD into composed Unicode NFC in a single non-overlapping regex pass. - Hierarchical Breadcrumb Chunker (
chunker.py): Tracks both Markdown headings (# H1 > ## H2 > ### H3) and Vietnamese legal hierarchy (Phần > Chương > Mục > Điều > Khoản), prepending[Ngữ cảnh: ...]to every chunk and extracting metadata (so_hieu,ngay_ban_hanh,loai_van_ban). - 3-Tier Hardware-Aware ONNX Acceleration (
onnx_embedder.py):- Tier 1 (Default):
CPUExecutionProvider(INT8,batch_size=32). - Tier 2 (Windows DirectML):
DmlExecutionProvider(FP32, boundedbatch_size=12for AMD Radeon RX 6600, RX 5000, Intel Arc, NVIDIA DX12) with warmup tensor check and automatic CPU fallback. - Tier 3 (Community Extras):
goldpan[cuda],goldpan[openvino],goldpan[coreml].
- Tier 1 (Default):
- Diagnostics & Migration:
goldpan doctortelemetry andgoldpan reindexto migrate v2goldpan_kb_localcollections into v3.
🛡️ 2. Pillar A — Agent Context Firewall (THREAT_MODEL.md)
- Three Operating Modes:
sanitize(default): Deterministically strips structural/steganographic injection vectors before tokenization without corrupting visible text.audit: Non-destructive inspection returningrisk_scoreandsignalsfor security audits (goldpan audit).strict: Rejects requests immediately withPromptInjectionBlockedErrorupon detecting injection signals.
- Multi-Vector Threat Defense:
- HTML/CSS DOM: Strips
display:none,visibility:hidden,opacity:0,font-size:0, white-on-white, off-screen positions,<template>, and suspicious HTML comments before Trafilatura parsing. - Document Binaries: Inspects PDF characters via
pdfplumberfor<1pt, white text, or off-page bounding boxes; inspects DOCX<w:vanish/>hidden runs. - Unicode Steganography: Strips Unicode Tags (
U+E0000–U+E007F), Bidi overrides, and Zero-Width characters. - Control Tokens & Beacons: Neutralizes fake delimiters (
<|im_start|>,[INST],<<SYS>>) and blocks Markdown image exfiltration beacons (). - Semantic Overrides: Scores English and Vietnamese imperative hijacking phrases with 0% False Positive Rate (FPR) on academic papers and OWASP LLM01 guides.
- HTML/CSS DOM: Strips
- MCP Provenance & Spotlighting: Attaches
_meta.provenance.firewalltelemetry and optional<untrusted_document_content>XML Spotlighting.
📊 3. Pillar C — Reproducible GoldPan Bench (BENCHMARK_REPORT.md)
Dual-track evaluation suite with 95% Wilson Score Confidence Intervals (goldpan bench --ci-gate):
Track B: Multilingual & Vietnamese Hybrid Retrieval (Held-out 16 queries)
- Recall@3:
BM25-only:62.5%([38.6%, 81.5%])v2.1 MiniLM+BM25 RRF:81.2%([57.0%, 93.4%])v3.0 E5+FTS5+Breadcrumb+VNI/TCVN3:93.8%([71.7%, 98.9%])
- Query Categories in v3.0:
vi_accented:100.0%Recall@1 & Recall@3 (MRR = 1.0000)vi_unaccented:100.0%Recall@1 & Recall@3 (MRR = 1.0000)vi_paraphrase:100.0%Recall@1 & Recall@3 (MRR = 1.0000)cross_lingual_en_vi:75.0%Recall@3
Track A: Agent Context Firewall (13 attacks + 4 hard negatives)
- Attack Detection Rate (TPR):
100.0%([77.2%, 100.0%]95% Wilson CI) - Payload Sanitization Rate:
100.0%([77.2%, 100.0%]95% Wilson CI) - Hard-Negative False Positive Rate (FPR):
0.0%([0.0%, 49.0%]95% Wilson CI) - Benign Content Preservation Rate:
100.0% - Firewall Overhead Latency:
0.21 ms/doc
📦 Installation
# Core package
pip install -e .
# With Windows DirectML GPU acceleration (AMD RX 6600, RX 5000, Intel Arc, NVIDIA DX12)
pip install -e ".[directml]"