1.39.3

1.39.3 (2025-10-06)

Fix

• fix: Only pin model name and rank (#3263)

Currently we pin 3 columns, this makes it hard or impossible to view on phones. The 3rd column is also no longer garuanteed as RTEB leaderboard does not use the zero-shot column (58a81a9)

• fix: Move zero-shot percentage calculation to the end of summary (#3231)

• Refactor: Move zero-shot percentage calculation to the end of summary table creation which only apply to RTEB table.

• Update RTEB benchmark name from "RTEB(beta)" to "RTEB" for consistency in display.

• feat - RTEB(beta)

• feat - remove Zero-shot

---

Co-authored-by: ethan <smiletoye@gmail.com> (65829bd)

Unknown

• model: Add ReasonIR (#3221)

• model: Add ReasonIR

• Update mteb/models/reasonir_model.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

• Update mteb/models/reasonir_model.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

• update n_parameters of ReasonIR

Co-authored-by: Niklas <n.muennighoff@gmail.com>

---

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Niklas <n.muennighoff@gmail.com> (f2504bd)

• fix bm25 on small datasets (#3261) (237d8dc)

• Update tasks & benchmarks tables (c8ae52c)

• Added Japanese to Retrieval (#3252)

• feat - add Japanese

• feat - use mteb.get_benchmark

• fix - 3.9 test error

• Revert "fix - 3.9 test error"

This reverts commit 6bfee53.

• fix - 3.9 test error (53b1c29)

• Fix AbsTaskTextRegression task (#3257)

Fix AbsTaskTextRegression (08b98cd)

• Update tasks & benchmarks tables (89bec7d)

• Aggregate by subset for HUMEv1 (#3255)

aggregate by subset for HUMEv1 (36901eb)

• fix python39 transformers compatibility (#3254)

• fix python39 transformers

• fix (85e1dd9)

• fix max tokens (#3243) (2f6eb2a)