Skip to content

2.18.10

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 30 Jul 18:10
· 332 commits to main since this release
2.18.10
214469a

2.18.10 (2026-07-30)

Ci

  • ci: test reference models on all benchmarks (#5030)

  • test reference models on all benchmarks

  • skip coderag

  • use benchmark/task combo

  • format citations (136b399)

Fix

  • fix: compute relevance scores in float32 to avoid low-precision ties (HPS) (#4933)

  • fix: compute relevance scores in float32 to avoid low-precision ties (HPS)

The similarity primitives in mteb.similarity_functions converted non-tensor
inputs to float32 but left existing tensors at their incoming dtype. When an
encoder returns float16/bfloat16 embeddings (e.g. models run in low precision),
cosine/dot scoring was therefore performed in low precision. Reduced-mantissa
formats coarsely bucket values in a narrow range, collapsing distinct relevance
scores into spurious ties; ranking then depends on arbitrary tie-breaking,
inflating retrieval-metric variance and bias.

Upcast float16/bfloat16 embeddings to float32 before scoring (High-Precision
Scoring). The upcast is value-preserving — every float16/bfloat16 value is
exactly representable in float32 — and a no-op for embeddings that are already
float32, so leaderboard results for full-precision models are unchanged.

Reference: "Reliable Evaluation Protocol for Low-Precision Retrieval"
(Yang et al., 2026), https://aclanthology.org/2026.acl-short.33

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

  • docs: shorten HPS comments in similarity primitives

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

  • Refactor tensor conversion in similarity_functions.py

Refactor tensor conversion to handle low-precision types more efficiently.

  • Refactor tensor conversion function

Removed low-precision upcasting for tensors.

  • Enhance tensor conversion for floating point types

Upcast sub-float32 floats to float32 in tensor conversion.

  • format

Co-authored-by: Kisu Yang <lab4@vaiv.kr>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (4b4f5de)

Unknown

  • model: Add BGE Small Structural Separator model implementation (#4928)

  • Add BGE Small Structural Separator model implementation

  • Load structural separator from full Hugging Face checkpoint

  • Address structural separator review feedback

  • Address remaining structural separator review feedback

  • Use standard encoder protocol (1019eed)

Assets 6
Loading