Feat: blockwise quantization recall restore#80
Merged
Conversation
… (Q8_0) - Re-route HNSW rebuilds in simple/source RAG to original f32 embeddings, recovering semantic search recall to 1.000 (resolving compound distortion loop). - Implement block-wise quantization (GGML Q8_0 style, 24 blocks of 32 elements) to reduce outlier sensitivity. - dynamic cosine similarity kernel supporting dynamic fallback for legacy 768-byte uniform and 864-byte block-wise blobs. - Platform-gate HNSW streaming rebuild for macOS default, opt-in elsewhere. - Bump versions to 0.20.0 (root) and 0.19.2 (rust_builder). - Add support for Windows/Linux platforms.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
PR Description: Restore HNSW Index Recall & Implement Block-wise Quantization
This PR restores the semantic search recall on the balanced wiki evaluation dataset (recovering from a degraded state back to
1.000) and integrates a backward-compatible, block-wise scalar quantization (Q8_0 style) pipeline.1. Core Problem & Resolution
A. Compound Distortion Loop (HNSW Recall Recovery)
vector_quant_i8feature was active, index rebuilding fetched the lossyi8quantized vectors and dequantized them in memory to construct thesource_rag.rsandsimple_rag.rsto load the original high-precisionchunks.embedding) directly from the database.B. Outlier Vulnerability (Block-wise Quantization)
C. Zero-Schema Database Migration & Backward Compatibility
chunks.embedding_i8blob (864 bytes for a 768-dim vector). The similarity kernel checks the blob's size at runtime:2. File-by-File Changes
Rust Core (
rag_engine_flutter/rust_builder)quantize_f32_to_u8_blobunder#[cfg(feature = "vector_quant_i8")]to output the 864-byte packed block-wise format.QueryQ8struct for precomputing block-wise quantized representations of search queries.cosine_similarity_q8similarity kernel featuring the backward compatibility gate.test_blockwise_cosine_similarityunit test verifying mathematical parity (embedding_i8dequantization.QueryQ8andcosine_similarity_q8inside the exact-scan pipeline.embedding_i8dequantization.QueryQ8andcosine_similarity_q8inside simple search paths.QueryQ8andcosine_similarity_q8inside the hybrid exact-scan loop.hnsw_streaming_rebuild.Packaging & Release
rust_builder/pubspec.yaml: Version bumped to0.19.2.pubspec.yaml(root): Version bumped to0.20.0and updatedrag_engine_flutterconstraint to^0.19.2.README.md&docs/guides/quick_start.md: Updated to show extended platform support (Windows & Linux).3. Verification & Benchmark Status
Core Unit Tests
All 154 Rust cargo unit tests pass:
cargo test --lib -- --test-threads=1On-Device Relevance Test (Flutter Drive macOS Profile)
All relevance tests pass:
default_hybridProfile: Semantic classpassage_recall@10is restored to1.000(100% recall parity).vector_onlyProfile: Semantic classpassage_recall@10is restored to0.750(up from degraded0.625in the uniform quantization build).