Skip to content

2.18.12

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 02 Aug 12:05
· 320 commits to main since this release
2.18.12
18f41f8

2.18.12 (2026-08-02)

Documentation

  • docs: add multiple choice retrieval task example (#4931)

Add a dedicated example to the 'Adding a Task' docs showing how to
implement a multiple choice retrieval task. It explains that such tasks
compare each query against a fixed candidate set via the top_ranked
split and use accuracy as the main score, and demonstrates building the
top_ranked split from relevant_docs in dataset_transform, following the
BLINK task pattern.

Closes #4408 (3748f63)

Fix

  • fix: Correctly use .n_active_parameters so that active parameters correct when using the API (#5066)

Active parameters is incorrect when using the API
Fixes #5064

@Samoed anything I am missing here? (ee5689e)

Unknown

  • Fix/afrie5 loader kwargs + changing AfriE5 revision (#5059)

  • fix: pass dtype via model_kwargs and drop invalid normalized kwarg for AfriE5-Large-instruct

  • fix: bump AfriE5-Large-instruct revision to include the modules.json,sentence_xlm-roberta_config.json, and 1_Pooling/config.json file additions

Co-authored-by: Nicolas Helmeyer <helmeyen@login-3.server.mila.quebec>
Co-authored-by: Nicolas Helmeyer <helmeyen@login-1.server.mila.quebec> (d0bad1e)

  • Add Hanno-Labs/dinghy-law-8b-v1 (legal embedding model) (#5058) (4ce470f)

  • Move data to mteb HF repo (mmvu, covers80, flare, miao, seavl, shs100k) (#5056) (0456b96)

  • benchmark: Expand MTEB(kor, v*) with retrieval/STS/NLI/clustering tasks (#4870)

  • Add 7 Korean retrieval tasks to MTEB(kor, v1)

Extend MTEB(kor, v1) retrieval coverage with LawIRKo, SQuADKorV1Retrieval,
AutoRAGRetrieval, PublicHealthQA, BelebeleRetrieval, MultiLongDocRetrieval,
and MrTidyRetrieval (kor subset), matching the KURE Korean retrieval set.

  • Add Korean STS/NLI/clustering tasks to MTEB(kor, v1)

Extend MTEB(kor, v1) with STS17 (ko-ko), KLUE-NLI and PawsXPairClassification
(NLI / pair classification), and SIB200ClusteringS2S, KlueMrcDomainClustering,
KlueYnatMrcCategoryClustering (clustering). Kor-NLI is omitted (no mteb task).

  • Add ModelMeta for 9 Korean community embedding models

Register dragonkue/BGE-m3-ko, dragonkue/multilingual-e5-small-ko,
dragonkue/snowflake-arctic-embed-l-v2.0-ko, exp-models/dragonkue-KoEn-E5-Tiny,
jhgan/ko-sroberta-multitask, nlpai-lab/KURE-v1, nlpai-lab/KoE5,
telepix/PIXIE-Rune-v1.5, upskyy/bge-m3-korean so their Korean results render on
the leaderboard. Metadata fetched from the HF Hub; loaders mirror each base family.

  • Support Korean (kor) in bm25s baseline tokenization

Add 'kor' to the bm25 language map with character-level tokenization (matching
the existing jpn/zho handling for no-space scripts), so the bm25s reference
baseline can index Korean retrieval tasks in MTEB(kor, v1).

  • Address review: MTEB(kor, v2), rename korean_models.py
  • Create MTEB(kor, v2) with the expanded task list instead of mutating
    MTEB(kor, v1). v1 is reverted to its original 6 tasks (Korean Legacy,
    alias 'MTEB(kor, classic)'); v2 holds the 19 tasks and takes the 'MTEB(kor)'
    alias. Leaderboard menu points the language-specific entry at v2 and keeps
    v1 under the legacy 'Other' section, mirroring MTEB(eng, v1/v2).
  • Rename korean_finetuned_models.py -> korean_models.py.

(n_embedding_parameters is already set on all 9 entries.)

  • Actually add MTEB(kor, v2) + menu registration (was missing from prior commit)

The prior commit only captured the file rename; the benchmarks.py / init.py /
_leaderboard_menu.py edits were not staged. This commit adds them:

  • MTEB(kor, v2) with the expanded task list; MTEB(kor, v1) reverted to its
    original 6 tasks (Korean Legacy, alias 'MTEB(kor, classic)').
  • Register MTEB_KOR_V2 in benchmarks init (import + all).
  • Leaderboard menu: v2 in language-specific, v1 in legacy 'Other'.
  • Fill training_datasets where documented (review follow-up)

Sources: each model's card at the pinned revision (+ KURE GitHub repo).

  • upskyy/bge-m3-korean, jhgan/ko-sroberta-multitask: trained on KorSTS(+KorNLI;
    no mteb task) -> {"KorSTS"}.
  • dragonkue/multilingual-e5-small-ko, exp-models/dragonkue-KoEn-E5-Tiny,
    dragonkue/snowflake-arctic-embed-l-v2.0-ko: AI Hub Korean MRC corpora only
    (none are mteb datasets) -> empty set, so base-model data is still inherited
    via adapted_from.
  • dragonkue/BGE-m3-ko, nlpai-lab/KURE-v1, telepix/PIXIE-Rune-v1.5: fine-tuning
    data unnamed in the card -> left None (unknown).
  • nlpai-lab/KoE5: trained on nlpai-lab/ko-triplet-v1.0 (linked via
    public_training_data) whose source composition is undocumented -> left None.

  • BGE-m3-ko: training_datasets set() (author-confirmed, no mteb data in fine-tune)

  • Move pixie_rune_v1_5 to pixie_models.py (review request)

Relocate the telepix/PIXIE-Rune-v1.5 ModelMeta from korean_models.py to
pixie_models.py alongside PIXIE-Rune-v1.0, reusing that file's
PIXIE_RUNE_V1_PROMPTS (identical query-prefix scheme). Entry values unchanged.

  • Fix max_tokens: 8194 -> 8192 for all affected entries

8194 is max_position_embeddings (incl. XLM-R's 2 offset slots), not the
usable input length: these models' tokenizer_config model_max_length and
sentence-transformers max_seq_length are both 8192, which is what
ModelMeta.max_tokens is documented to mean. Fixes the three Korean bge-m3
fine-tunes (per review suggestion) and the same error inherited in
BAAI/bge-m3 itself, manu/bge-m3-custom-fr, GreenNode VN x2,
AITeamVN/Vietnamese_Embedding, deepvk/USER-bge-m3, jina-embeddings-v3.

  • Add KorNLI to MTEB(kor, v2)

KorNLI is now available in mteb; include it in the v2 pair-classification
set (v2: 19 -> 20 tasks). Merged main to pick up the task definition.

  • Move new model registrations out to #4921 (review request)

Per review, the 9 Korean community ModelMetas now live in #4921 so that
embeddings-benchmark/results#581 can merge independently; this PR stays a
benchmark-definition change (MTEB(kor, v2), bm25s Korean tokenization,
max_tokens fixes).

  • MTEB(kor): mark v1 superseded by v2, describe v1->v2 changes

Address review on #4870:

  • MTEB(kor, v1): aliases=[], display_name="Korean", add superseded_by=["MTEB(kor, v2)"]
  • MTEB(kor, v2): expand description to state what changed since v1 (grows 6 -> 20 tasks)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-authored-by: dgyu <dgyu@sionic.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> (db49990)

  • model: Add dragonkue/multilingual-e5-small-ko-v2 and sionic-ai/comsat-e… (#5054)

  • Register dragonkue/multilingual-e5-small-ko-v2 and sionic-ai/comsat-embed-ko-8b-preview

Two Korean community embedding models evaluated on MTEB(kor, v2):

  • dragonkue/multilingual-e5-small-ko-v2 (multilingual-e5-small finetune)
  • sionic-ai/comsat-embed-ko-8b-preview (Qwen3-Embedding-8B finetune)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

  • Address review: mark e5-small-ko superseded, trim comsat-ko comment
  • dragonkue/multilingual-e5-small-ko: add superseded_by=multilingual-e5-small-ko-v2
  • sionic-ai/comsat-embed-ko-8b-preview: remove redundant loader comment (per review)

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> (53a9c40)

  • Add perplexity-ai/pplx-embed-v1-late-0.6b model meta (#4813)

  • Add perplexity-ai/pplx-embed-v1-late-0.6b model meta

Closes #4691. Adds the ModelMeta entry for perplexity-ai/pplx-embed-v1-late-0.6b,
a PyLate/ColBERT late-interaction (MaxSim) embedding model with 128-dim
token-level vectors, continued-trained from perplexity-ai/pplx-embed-v1-0.6b.
Metadata sourced from the HF model card/API and config files (revision, MIT
license, 128-dim projection, 512 document length, ~596M params).

  • Add embedding parameter count for pplx late embed (7c2712d)

  • dataset: Repoint dead dataset to mteb's supported for AfriSentiLangClassification (#4912)

  • Fixed AfriSentiLangClassification: repointed dead HausaNLP dataset to mteb/afri_senti_lang mirror, removed redundant tweet->text preprocessing step

  • remove processing

Co-authored-by: Nicolas Helmeyer <helmeyen@login-2.server.mila.quebec>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (74044b9)

Contributors

Samoed
Assets 6
Loading