Skip to content

GoldPan AI v3.0.0 — Multilingual E5 ONNX RAG, Agent Context Firewall & Dual-Track Benchmarks

Choose a tag to compare

@ptai-eng ptai-eng released this 31 Oct 07:20
· 9 commits to main since this release

🚀 What's New in GoldPan AI v3.0.0

GoldPan AI v3.0.0 marks a major architectural milestone, evolving from a multimodal converter into a hardened, privacy-first data refinery, Multilingual E5 ONNX + SQLite FTS5 Hybrid RAG engine, and Agent Context Firewall for autonomous LLM agents (MCP, REST API, CLI).


🏛️ 1. Pillar B — Multilingual & Vietnamese First-Class Core

  • intfloat/multilingual-e5-small ONNX Embeddings (384-dim): Automatic passage: prefix on document ingestion, query: prefix on search, and L2 normalization in isolated collection goldpan_kb_e5_v3.
  • Persistent SQLite FTS5 Lexical Index (chroma_data/goldpan_fts.db): Replaces RAM-heavy in-memory BM25 with persistent disk-backed FTS5 (WAL mode) indexing both accented text_nfc and ASCII-folded text_folded (đ/Đ -> d + compound bigrams) fused via Reciprocal Rank Fusion (RRF, k=60).
  • Legacy Vietnamese Encoding Decoder (vn_encoding.py): Automatically detects and converts VNI-Windows, TCVN3 (ABC), and decomposed Unicode NFD into composed Unicode NFC in a single non-overlapping regex pass.
  • Hierarchical Breadcrumb Chunker (chunker.py): Tracks both Markdown headings (# H1 > ## H2 > ### H3) and Vietnamese legal hierarchy (Phần > Chương > Mục > Điều > Khoản), prepending [Ngữ cảnh: ...] to every chunk and extracting metadata (so_hieu, ngay_ban_hanh, loai_van_ban).
  • 3-Tier Hardware-Aware ONNX Acceleration (onnx_embedder.py):
    • Tier 1 (Default): CPUExecutionProvider (INT8, batch_size=32).
    • Tier 2 (Windows DirectML): DmlExecutionProvider (FP32, bounded batch_size=12 for AMD Radeon RX 6600, RX 5000, Intel Arc, NVIDIA DX12) with warmup tensor check and automatic CPU fallback.
    • Tier 3 (Community Extras): goldpan[cuda], goldpan[openvino], goldpan[coreml].
  • Diagnostics & Migration: goldpan doctor telemetry and goldpan reindex to migrate v2 goldpan_kb_local collections into v3.

🛡️ 2. Pillar A — Agent Context Firewall (THREAT_MODEL.md)

  • Three Operating Modes:
    • sanitize (default): Deterministically strips structural/steganographic injection vectors before tokenization without corrupting visible text.
    • audit: Non-destructive inspection returning risk_score and signals for security audits (goldpan audit).
    • strict: Rejects requests immediately with PromptInjectionBlockedError upon detecting injection signals.
  • Multi-Vector Threat Defense:
    • HTML/CSS DOM: Strips display:none, visibility:hidden, opacity:0, font-size:0, white-on-white, off-screen positions, <template>, and suspicious HTML comments before Trafilatura parsing.
    • Document Binaries: Inspects PDF characters via pdfplumber for <1pt, white text, or off-page bounding boxes; inspects DOCX <w:vanish/> hidden runs.
    • Unicode Steganography: Strips Unicode Tags (U+E0000–U+E007F), Bidi overrides, and Zero-Width characters.
    • Control Tokens & Beacons: Neutralizes fake delimiters (<|im_start|>, [INST], <<SYS>>) and blocks Markdown image exfiltration beacons (![...](https://...?secret=...)).
    • Semantic Overrides: Scores English and Vietnamese imperative hijacking phrases with 0% False Positive Rate (FPR) on academic papers and OWASP LLM01 guides.
  • MCP Provenance & Spotlighting: Attaches _meta.provenance.firewall telemetry and optional <untrusted_document_content> XML Spotlighting.

📊 3. Pillar C — Reproducible GoldPan Bench (BENCHMARK_REPORT.md)

Dual-track evaluation suite with 95% Wilson Score Confidence Intervals (goldpan bench --ci-gate):

Track B: Multilingual & Vietnamese Hybrid Retrieval (Held-out 16 queries)

  • Recall@3:
    • BM25-only: 62.5% ([38.6%, 81.5%])
    • v2.1 MiniLM+BM25 RRF: 81.2% ([57.0%, 93.4%])
    • v3.0 E5+FTS5+Breadcrumb+VNI/TCVN3: 93.8% ([71.7%, 98.9%])
  • Query Categories in v3.0:
    • vi_accented: 100.0% Recall@1 & Recall@3 (MRR = 1.0000)
    • vi_unaccented: 100.0% Recall@1 & Recall@3 (MRR = 1.0000)
    • vi_paraphrase: 100.0% Recall@1 & Recall@3 (MRR = 1.0000)
    • cross_lingual_en_vi: 75.0% Recall@3

Track A: Agent Context Firewall (13 attacks + 4 hard negatives)

  • Attack Detection Rate (TPR): 100.0% ([77.2%, 100.0%] 95% Wilson CI)
  • Payload Sanitization Rate: 100.0% ([77.2%, 100.0%] 95% Wilson CI)
  • Hard-Negative False Positive Rate (FPR): 0.0% ([0.0%, 49.0%] 95% Wilson CI)
  • Benign Content Preservation Rate: 100.0%
  • Firewall Overhead Latency: 0.21 ms/doc

📦 Installation

# Core package
pip install -e .

# With Windows DirectML GPU acceleration (AMD RX 6600, RX 5000, Intel Arc, NVIDIA DX12)
pip install -e ".[directml]"