Skip to content

v1.3.0 — Reasoning-Intensive Retrieval & BRIGHT 12-Split SOTA

Latest
Latest

Choose a tag to compare

View all tags
@hemanth hemanth released this 03 Nov 14:52
v1.3.0
ded1d08

Highlights in v1.3.0

  • Reasoning-Intensive Retrieval: Added ThinkThenRetrieve (use_think_then_retrieve=True), CausalLLMReranker (reranker_type="causal_llm"), and ListwiseReasoningReranker (reranker_type="listwise").
  • Official BRIGHT (xlangai/BRIGHT, ICLR 2025) 12-Split GPU Evaluation:
    • 32.61% Macro NDCG@10 (95% CI [31.01%, 34.67%], Recall@100 = 71.98%, 2.87x BM25) with standardized CoT + Hybrid RRF + Rocchio PRF + BAAI/bge-reasoner-embed-qwen3-8b-0923.
    • 31.43% Macro NDCG@10 (95% CI [30.14%, 33.81%], 2.77x BM25) 100% open-weights using local Qwen2.5-7B-Instruct ThinkThenRetrieve.
    • 44.90% NDCG@10 on theoremqa_theorems (9.1x BM25) and 41.24% NDCG@10 on theoremqa_questions.
  • Official AIR-Bench 24.05 Long-Doc Leaderboard: Ranked #1 & #2 Overall in the World (84.96% Recall@10, 75.72% nDCG@10, 80.10% MRR@10).
  • Thread-Safe Parallel Telemetry & Boundary Hardening: Fixed concurrent contextvars.Context propagation (ctx.copy().run(...)) across ThreadPoolExecutor query variations and guarded top_k <= 0.
Assets 5
Loading