Skip to content

2.7.29

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 12 Feb 14:55
· 999 commits to main since this release
2.7.29
7d4cc63

2.7.29 (2026-02-12)

Documentation

  • docs: Improved adding a benchmark docs (#4087)

  • docs: Improved adding a benchmark docs

expanded the explanation of provide more information about the process.

  • Update docs/contributing/adding_a_benchmark.md

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

  • Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

  • fix

  • fix

  • minor heading change

  • Apply suggestions from code review

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (8149d4a)

Fix

  • fix: constrain the transformers library version for jina-clip (#4061)

  • fix: constrain the transformers library version for jina-clip to avoid compatibility issue

  • add require package

  • ad to conflicts

  • try to run again

  • upd lock

  • tmp

  • try

  • fix: pylate dependency on outdated version of transformers

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (eee82cf)

Unknown

  • dataset: Add MTEB(spa) Spanish language benchmark (#4053)

  • dataset: Add MTEB(spa) Spanish language benchmark

Define MTEB(spa, v1) benchmark grouping 23 existing Spanish tasks
across 6 task types: Classification (8), Clustering (3),
PairClassification (2), Reranking (1), Retrieval (5), and STS (4).

  • fix: Replace MIRACLRetrieval with HardNegatives.v2 per review

  • fix: Remove tasks with known issues, add contact, reduce to 16 tasks

  • Apply suggestion from @KennethEnevoldsen

Co-authored-by: Clemente <clemente@Clementes-MacBook-Pro.local>
Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me> (2507bec)

  • Move MIEB datasets to mteb HuggingFace org (#4070)

  • Move 15 MIEB datasets to mteb HuggingFace org

Update dataset paths and revisions for tasks that now use datasets
forked to the mteb org:

  • MMSoc_HatefulMemes, MMSoc_Memotion (Ahren09 -> mteb)
  • blink-it2i, blink-it2i-multi, blink-it2t, blink-it2t-multi (JamieSJS -> mteb)
  • gld-v2-i2t, imagecode, imagecode-multi (JamieSJS -> mteb)
  • imagenet-10, imagenet-dog-15, met (JamieSJS -> mteb)
  • r-oxford-easy-multi, r-oxford-medium-multi, r-oxford-hard-multi (JamieSJS -> mteb)

Part of #4049

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • transfer mrbench

  • Move 8 isaacchung MIEB datasets to mteb HuggingFace org

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • Move 12 MIEB datasets to mteb HuggingFace org

Datasets moved:

  • dpdl-benchmark/sun397
  • ethz/food101
  • floschne/xflickrco
  • floschne/xm3600
  • flwrlabs/ucf101
  • nyu-visionx/CV-Bench
  • tanganke/dtd
  • tanganke/stl10
  • timm/eurosat-rgb
  • timm/resisc45
  • uoft-cs/cifar10
  • uoft-cs/cifar100

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • Move 15 MIEB datasets to mteb HuggingFace org

Datasets moved:

  • JamieSJS/r-paris-easy-multi
  • JamieSJS/r-paris-medium-multi
  • JamieSJS/r-paris-hard-multi
  • JamieSJS/rp2k
  • JamieSJS/sketchy
  • JamieSJS/stanford-online-products
  • JamieSJS/vizwiz
  • JamieSJS/vqa-2
  • Pixel-Linguist/rendered-sts12
  • Pixel-Linguist/rendered-sts13
  • Pixel-Linguist/rendered-sts14
  • Pixel-Linguist/rendered-sts15
  • Pixel-Linguist/rendered-sts16
  • ylecun/mnist
  • zh-plus/tiny-imagenet

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • Move 7 clip-benchmark datasets to mteb HuggingFace org

Datasets moved:

  • clip-benchmark/wds_country211
  • clip-benchmark/wds_fer2013
  • clip-benchmark/wds_gtsrb
  • clip-benchmark/wds_renderedsst2
  • clip-benchmark/wds_vtab-clevr_closest_object_distance
  • clip-benchmark/wds_vtab-clevr_count_all
  • clip-benchmark/wds_vtab-pcam

Note: wds_imagenet1k failed due to storage limits.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • Move 14 MIEB datasets to mteb HuggingFace org

Datasets migrated:

  • clip-benchmark/wds_imagenet1k → mteb/wds_imagenet1k
  • m-a-p/SciMMIR → mteb/SciMMIR
  • yjkimstats/SUGARCREPE_fmt → mteb/SUGARCREPE_fmt
  • nelorth/oxford-flowers → mteb/oxford-flowers
  • vidore/arxivqa_test_subsampled_beir → mteb/arxivqa_test_subsampled_beir
  • vidore/docvqa_test_subsampled_beir → mteb/docvqa_test_subsampled_beir
  • vidore/infovqa_test_subsampled_beir → mteb/infovqa_test_subsampled_beir
  • vidore/shiftproject_test_beir → mteb/shiftproject_test_beir
  • vidore/syntheticDocQA_artificial_intelligence_test_beir → mteb/syntheticDocQA_artificial_intelligence_test_beir
  • vidore/syntheticDocQA_energy_test_beir → mteb/syntheticDocQA_energy_test_beir
  • vidore/syntheticDocQA_government_reports_test_beir → mteb/syntheticDocQA_government_reports_test_beir
  • vidore/syntheticDocQA_healthcare_industry_test_beir → mteb/syntheticDocQA_healthcare_industry_test_beir
  • vidore/tabfquad_test_subsampled_beir → mteb/tabfquad_test_subsampled_beir
  • vidore/tatdqa_test_beir → mteb/tatdqa_test_beir

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

  • update rest

Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (2ef04f8)

  • model: add voyage-4-nano (#4086)

  • model: add voyage-4-nano model implementation

  • Apply suggestion from @Samoed

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

Co-authored-by: Kenneth Enevoldsen <kenevoldsen@pm.me>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (2ce07c4)

Contributors

KennethEnevoldsen and Samoed
Assets 6
Loading