Reranker 7-lane + ONNX correction tables + Cronbach quorum#140
Merged
Conversation
Reranker v3: cos[-0.886, 0.826] E/I=50.2% (balanced inhibition!) Lanes 6-7 computed from existing cosine matrix. Cronbach alpha cross-model: ALL lanes < 0.37 (NO agreement) → Confirms: cross-model calibration impossible (each codebook unique) → Spearman rho: 0.01-0.07 between models (essentially zero) Intra-model lane consistency: Qwen3-VL: alpha=0.605 (lanes see different things → superposition valuable) Jina v5: alpha=0.611 (same) Reranker: alpha=0.920 (lanes redundant, strong balanced signal) Key insight: Reranker has 50% negative pairs (real inhibition). Embedding models have 99%+ positive (no inhibition). Combine both for thinking engine: embeddings for discrimination, reranker for inhibition. https://claude.ai/code/session_019RzHP8tpJu55ESTxhfUy1A
Reranker delta = where embedding model is WRONG: 28% of Qwen3-VL pairs get strong inhibition (delta < -0.3) 37% of Jina v5 pairs get strong inhibition Correction tables (f32 + quantized i8): Per centroid pair: reranker_cos - embedding_cos alpha=0.5 blend doubles discrimination range (std 0.064 → 0.123) The 20KB ONNX learns to predict this correction at runtime without needing the reranker model. Inhibition baked in. https://claude.ai/code/session_019RzHP8tpJu55ESTxhfUy1A
Removed unused imports: sensor.rs: CODEBOOK_SIZE (const never referenced) cognitive_trace.rs: DissonanceProfile (struct exists but unused here) contract_bridge.rs: VolitionalAct (designed, not yet wired) awareness_dto.rs: CouncilWeights (designed, not yet wired) world_model.rs: RungLevel (used in contract_bridge, not here) role_tables.rs: StackedN, f32_to_bf16 (only bf16_to_f32 needed) ground_truth.rs: TokenizerRegistry (calibration path incomplete) Fixed unused variables (computed but not yet consumed): contract_bridge.rs: free_energy → _free_energy (cascade termination TODO) world_model.rs: p_dist → _p_dist (family distance, gestalt TODO) domino.rs: threshold_3sigma → _threshold_3sigma (3σ outlier detection TODO) superposition.rs: count → _count, resonant_fraction → _resonant_fraction pooling.rs: mean_energy → _mean_energy (future gating) spiral_segment.rs: seg → _seg (only error used for boundary detection) tensor_bridge.rs: t → _t (Tensor variant conversion) Cleaned signed_domino.rs: removed 12 unused causal edge channel imports. The cascade engine code is valid but channels aren't wired yet. https://claude.ai/code/session_019RzHP8tpJu55ESTxhfUy1A
AdaWorldAPI
pushed a commit
that referenced
this pull request
Apr 17, 2026
…eam #150 ## Stale artifact removal (182 files, 3 MB) `AdaWorldAPI-lance-graph-d9df43b/` was a committed snapshot of an older upstream version (48 .rs files vs our 98). Full audit confirmed: - ZERO files exist only in the artifact (every file has a counterpart) - Every differing file: ours >= artifact in LOC (ours is strictly ahead) - All upstream features (#125 parameter_substitution, #140 lance_vector_search) are already in our src tree The directory created GitHub path confusion — duplicate navigation paths for datafusion_planner, spo, blasgraph, neighborhood, arigraph. Removing it eliminates that confusion with zero content loss. ## Cherry-pick: spark_dialect.rs from upstream PR #150 The ONE file upstream has that we didn't: - `crates/lance-graph/src/spark_dialect.rs` (107 LOC) Spark SQL dialect for DataFusion unparser: backtick quoting, STRING type casting, EXTRACT for dates, BIGINT/INT types, LENGTH(), derived table aliases. - `crates/lance-graph/tests/test_to_spark_sql.rs` (293 LOC) Full test suite for Spark SQL output. - `pub mod spark_dialect;` added to lib.rs Adapted from upstream's DF 50.3 to our DF 51 — same API surface, no changes needed. ## Upstream audit result (for the record) Upstream (lance-format/lance-graph) is at v0.5.4. Our fork is at v0.5.3 with newer deps (arrow 57 vs 56.2, datafusion 51 vs 50.3). Other than spark_dialect, every upstream feature and fix is already present in our source tree — parameter_substitution (#125), lance_vector_search (#140), complex RETURN clauses (#142), duplicate columns fix (#128) are all in `crates/lance-graph/src/`. Their deleted `simple_executor` was a prototype cold-path executor we never had. Our `ExecutionStrategy::DataFusion` path (6K LOC planner) + `ExecutionStrategy::BlasGraph` (semiring algebra) subsume it. The user has flagged adding a deliberate `ExecutionStrategy::Simple` cold path as a 4th strategy for trivial queries — that's a separate PR per the documented matrix of execution strategies. https://claude.ai/code/session_01NYGrxVopyszZYgLBxe4hgj
4 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Cronbach Alpha Results
Reranker as ONNX Training Signal
28-37% of pairs get strong inhibition (delta < -0.3).
64 KB i8 correction table per model = ONNX training target.
Files
releases/v0.2.0-7lane-codebooks/jina-reranker-v3-BF16-7lane/— full 7-lane encodingreleases/v0.2.0-7lane-codebooks/*/reranker_correction_256x256.{f32,i8}— correction tableshttps://claude.ai/code/session_019RzHP8tpJu55ESTxhfUy1A