Skip to content

2.20.6

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 05 Sep 09:55
· 140 commits to main since this release
2.20.6
f5f1336

2.20.6 (2026-09-05)

Fix

  • fix: update stale tokenizer revisions for Transformers v5 (#5377)

  • fix: update stale tokenizer revisions

  • test: remove brittle tokenizer fingerprints (dd3cd65)

Unknown

  • Enables ANN205 (missing-return-type-static-method) Ruff Rule (#5380) (2e54d27)

  • Enables ANN (flake8-simplify) Ruff Rule (#5372)

  • Enables SIM (flake8-simplify) Ruff Rule

  • enable ANN in mteb/models folder

  • enable ANN in mteb/tasks folder (56fa292)

  • model: improve MIRACL query prompt for voyage-4-large (#5374)

Evolutionary prompt search over the 18-language macro. nDCG@10 0.65193 -> 0.67583,
better on all 18 languages, validated on 6 languages held out of the search.

Co-authored-by: fzowl <zoltan@voyageai.com> (d684be9)

  • model: add BioVITA multimodal encoder (#5346)

  • model: add BioVITA multimodal encoder

  • fix: address BioVITA model review comments

  • Delete mteb_mock_run_results.md

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (ae93300)

  • dataset: add Omnilingual ASR speech-text retrieval (a2t, t2a, 50 new languages) (#5365)

dataset: add Omnilingual ASR speech-text retrieval (50 new languages)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (f2abe47)

  • dataset: add GLAMI-1M retrieval (#5331)

  • dataset: add GLAMI-1M multimodal classification

  • dataset: address GLAMI-1M review feedback

  • dataset: keep GLAMI names and descriptions separate

  • dataset: add GLAMI-1M image-to-text retrieval

  • Rename glami_1m_t2i_retrieval.py to glami_1m_retrieval.py

  • Fix GLAMI retrieval import after rename

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (e217c5a)

  • dataset: add CAMEO multilingual speech emotion classification (a2c, 5 languages) (#5366)

dataset: add CAMEO multilingual speech emotion classification (5 languages)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (e253d04)

  • task: add Lombard GRID audiovisual retrieval tasks (#5337)

  • task: add Lombard GRID i2va retrieval

  • task: add Lombard GRID audiovisual retrieval tasks (4306a1b)

  • dataset: add Afri-MCQA multilingual speech-image retrieval (a2i, i2a) (#5356)

dataset: add Afri-MCQA multilingual speech-image retrieval (16 African languages)

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (fbda84a)

  • [MOEB] Add XModBench any-to-any retrieval tasks (#5294)

  • Add XModBench any-to-any retrieval tasks

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • feat: change to reranking

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • refactor: restore extract datasets

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • feat: per direction language metadata

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • Update mteb/tasks/aggregated_tasks/eng/xmod_bench.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

  • Update mteb/tasks/aggregated_tasks/eng/xmod_bench.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

  • style: lint error

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • refactor: remove aggregate

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

  • feat: add category to each row

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>

Signed-off-by: jupyterjazz <saba.sturua@jina.ai>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (b287699)

  • add kazalbrur/bangla-embed-e5-small-banglish (#5354)

feat(models): add kazalbrur/bangla-embed-e5-small-banglish

Bengali/Banglish sentence encoder (118M) distilled from BGE-M3 on top of
intfloat/multilingual-e5-small, with cross-script (romanized Bengali)
retrieval support. Uses the E5 query:/passage: prompt convention and a
1024-dim dense projection head.

Co-authored-by: Kazal Chandra Barman <kazal.chandra@technonext.com> (3334caa)

  • model: register nlpai-lab/KURE-v2 Korean ColBERT model (#5353)

feat: register nlpai-lab/KURE-v2 Korean ColBERT model

Add the ModelMeta for nlpai-lab/KURE-v2, a PyLate ColBERT (late-interaction,
multi-vector) model adapted from skt/A.X-Encoder-base, using the
MultiVectorModel loader. Training data is lightonai/embeddings-fine-tuning
plus a subsample of lightonai/embeddings-pre-training-curated.

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> (8dc44a3)

  • dataset: add AVCaps audio-visual retrieval (6 directions) (#5344)

Adds a2t/t2a, v2t/t2v and va2t/t2va over AVCaps, plus descriptive statistics.

AVCaps captions each clip three ways - audio only, visuals only, and both
together - so the audio-only, video-only and combined directions can be scored
independently on identical clips rather than inferred from one mixed caption set.

Audio is demuxed to a separate mono 16 kHz stream and each task exposes only the
media its captions describe, so an audio-caption task cannot be solved off the
video track. Official test split only: 200 clips.

Repeated caption text is dropped (725/788/808 -> 715/780/802). Annotators
sometimes write the same short sentence for different clips, which would leave
an identical query marked relevant to only one of the clips it describes.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (a1268f9)

  • Add CrisisMMD image-text classification tasks (#5340)

  • Add CrisisMMD image-text classification tasks

  • Add CrisisMMD zero-shot classification tasks

  • Revert "Add CrisisMMD zero-shot classification tasks"

This reverts commit 6123c90.

  • Inline CrisisMMD classification settings (12aab77)

  • dataset: add MVL-SIB sentence-to-image retrieval (#5334) (8487c1a)

  • fix: Make it such that torch is not a required import of mteb.types (#5329)

  • make mteb.types torch-free

  • changes from review

  • rename file to tests/test_ensure_no_torch_at_import.py (5f1167a)

Assets 6
Loading