Skip to content

2.18.9

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 30 Jul 17:51
· 337 commits to main since this release
2.18.9
598abc6

2.18.9 (2026-07-30)

Fix

  • fix: upcast reranker logits to float32 before softmax (HPS) (#4934)

  • fix: upcast reranker logits to float32 before softmax (HPS)

The Qwen3 and monoT5-family cross-encoder rerankers apply log_softmax
directly to logits produced by a model running in bfloat16/float16. The
reduced mantissa of these formats coarsely buckets the (0, 1) probability
range, so distinct query-document relevance scores collapse into spurious
ties. Retrieval/reranking metrics then depend on arbitrary tie-breaking,
which inflates their variance and bias (e.g. up to ~38%p MRR@10 range and
+9.08%p bias for Qwen3-Reranker under bfloat16).

Apply High-Precision Scoring (HPS): upcast the logits to float32 immediately
before the softmax, leaving the forward pass in low precision. The upcast is
value-preserving, adds negligible cost, and restores near-full-precision
ranking stability.

Reference: "Reliable Evaluation Protocol for Low-Precision Retrieval"
(Yang et al., 2026), https://aclanthology.org/2026.acl-short.33

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

  • Update mteb/models/model_implementations/qwen3_reranker.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

  • docs: shorten HPS comments in monoT5 rerankers to match qwen3

Co-authored-by: Kisu Yang <lab4@vaiv.kr>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (ce4fb38)

Unknown

  • [MOEB] Add dataset SHS100K (#5045)

  • [MOEB] Add dataset SHS100K

  • fix first results issue

  • fix spelling lint issue (6d97b70)

  • model: Add mDenseOn/mLateOn definitions (#5048) (0b75fb8)

  • [MOEB] Add several FLARE-based tasks (#4991)

  • [MOEB] Add several FLARE-based tasks

  • formatting

  • fix description

  • add the descriptive stats

  • fix URL

  • fix citation

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (aadb54b)

  • [MOEB] Add MIAO-based datasets (#5001)

  • [MOEB] Add MIAO-based datasets

  • add descriptive stats

  • edit create data

  • update description

  • pin dataset version (db53995)

  • model: Add Bekko embedding models (#5043)

  • feat(models): add Bekko embedding models

  • docs(models): add Bekko paper citation

  • fix(models): use Bekko public release date

  • docs(models): clarify Bekko language metadata

  • feat(models): register Bekko core languages (ccd0914)

Assets 6
Loading