Skip to content

2.20.13

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 14 Sep 05:34
· 83 commits to main since this release
2.20.13
908b184

2.20.13 (2026-09-14)

Fix

  • fix: declare ModelMeta load dtypes with OutputDType instead of torch.dtype (#5441)

  • fix: declare ModelMeta load dtypes with OutputDType instead of torch.dtype

  • address copilot comments

  • address remaining copilot review comments

  • clarify OutputDType docstring: not all members map to a same-named
    torch dtype (INT4/UINT4/BINARY are compression-only levels)
  • note in compress_embeddings.md that the ValueError for an invalid
    output_dtype is raised at encode() time, not construction
  • fix AST guard in test_ensure_no_torch_at_import.py to track function
    body nodes by identity instead of line number, so a one-line function
    signature default is no longer mistaken for its body
  • broaden the guard's dtype whitelist to every torch.<name> attribute
    that is a torch.dtype, catching aliases like torch.half/torch.long
  • lint and format

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (c4f968c)

Unknown

  • dataset: add OmniWikiRetrieval v2i and v2t from Tevatron Wiki-SS-NQ (#5415)

dataset: add OmniWikiRetrieval (1443790)

  • move Nano retrieval tasks to the v2 dataset format (#5452)

refactor: move Nano retrieval tasks to the v2 dataset format

All 13 Nano* tasks carried an identical v1 load_data() override that read the
zeta-alpha-ai copies into self.corpus/self.queries/self.relevant_docs. The
re-uploads under the mteb org are in the v2 layout, so the override can go and
the default AbsTaskRetrieval loader handles them.

Verified per task that the v2 dataset loads identical content to the original:
corpus ids and text, query ids and text, and qrels all match for 13/13. The v1
override hardcoded every relevance score to 1; the uploaded qrels also carry
score 1 throughout, so scores are unchanged.

Refs #3424

Co-authored-by: Claude Opus 5 <noreply@anthropic.com> (f18a069)

  • Add tencent/EVIE-8B and tencent/EVIE-4.5B (#5450)

  • Add tencent/EVIE-8B and tencent/EVIE-4.5B

Co-authored-by: Cursor <cursoragent@cursor.com>

  • Fix ruff literal-membership in EvieWrapper

Co-authored-by: Cursor <cursoragent@cursor.com>

  • Declare evie extra as conflicting and update uv.lock

evie pins transformers>=5.13.1, which is incompatible with visrag-ret and
other extras (transformers<4.53). Add evie to [tool.uv] conflicts so uv can
resolve, and regenerate the lockfile.

Co-authored-by: Cursor <cursoragent@cursor.com>

  • Address review: optional FA2, simpler bidir call, drop flash-attn from evie extra
  • FA2 now optional: warn and fall back to attn_implementation='sdpa' when
    flash_attention_2 is unavailable, instead of raising.
  • Call self.model.enable_bidirectional_attention() directly; drop the
    getattr/callable guard and the manual _enable_bidirectional_attention helper.
  • Remove flash-attn from the evie extra (FA2 is an optional perf add-on) and
    regenerate uv.lock.

Co-authored-by: Cursor <cursoragent@cursor.com>

  • Delete mteb_mock_run_results.md

  • fix typing

Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a25a026)

  • dataset add ColDeRReranking benchmark (#5409)

  • feat(tasks): add ColDeRReranking benchmark (#2709)

  • test: add duplicate_text exemption for ColDeRReranking in task quality tests

  • fix(reranking): clarify ColDeR benchmark semantics

  • fix: normalize ColDeR citation formatting

  • fix(reranking): remove unused ColDeR logger

  • Apply suggestion from @Samoed

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com> (5427a1f)

  • model: Add litillabs/litil-embed-0.6b (#5456)

  • Add litillabs/litil-embed-0.6b

  • Remove mock run report (40a77d2)

  • model: Add models multi-modal-embed [MOEB] (#5011)

  • [MOEB] Add models multi-modal-embed

  • adding n_embedding parameter

  • fix large model

  • fix lint

  • robustness auto model or model

  • lock update + remove numpy cast + shared base for large and small + remove _COMMON

  • remove librosa, cherry pick custom code from large model, and remove patches

  • remove labels

  • simplifying implementation and pinning revisions

  • fix lint

  • update number of parameters (ef8b40f)

  • task: add COCO Modality Equivalence retrieval tasks (#5384)

  • feat: add COCO Modality Equivalence retrieval tasks (issue #5358)

  • fix: bibtex formatting and ruff format in coco modality equivalence

  • Sort bibtex entries alphabetically and add blank lines between them
    so bibtexparser round-trip matches the test expectation
  • Collapse multi-line author fields to single line (bibtexparser strips
    continuation indentation, breaking the equality check)
  • Reformat chained cast_column calls in create_data.py to satisfy
    ruff format

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

  • fix: correct BEIR format and per-config dataset routing
  • Each task class now specifies its own dataset config name (t2i, a2i_h,
    a2i_s, i2t, i2a_h, i2a_s) via _dataset() helper; the shared _COMMON
    dict no longer contains the dataset key
  • Rewrite create_data.py to load sources using the BEIR corpus/queries/qrels
    config structure; fix corpus split name for mbeir_mscoco_task0 ("corpus"
    not "test"); build all six BEIR-format DatasetDicts and save locally
    before push

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

  • Fill dataset revision and fix build script for COCO modality equivalence

    • Set _DATASET_PATH to rakshi719/coco-modality-equivalence
    • Set _DATASET_REVISION to 3aabaa047c92980e12628149df52425757b3f8a6
    • Fix Image/Audio decode=False to avoid PIL/torchcodec encoding issues
    • Fix ID normalization: extract 12-digit COCO IDs from image paths
    • Fix MSCOCO loading: use 'query' config (not 'queries') for text captions
    • Fix push format: 18 HF configs ({task}-corpus/queries/qrels) matching MTEB loader
    • Add Audio.encode_example monkeypatch for datasets 5.0.0 torchcodec issue

  • Fix ruff formatting in create_data.py

  • Add COCO modality equivalance task, analysis script and gap results

  • Add directional asymmetry analysis script and results (issue #5360)

  • Add sampling budget analysis script and results (issue #5362)

  • feat(analysis): PCA latent dimensions analysis for MTEB issue #5367

  • Fix COCO modality-equivalence retrieval tasks and add descriptive stats

  • Add load_data override so the HF config prefix (t2i/a2i_h/a2i_s/i2t/i2a_h/i2a_s)
    is actually used instead of always requesting the default config.
  • Fix create_data.py: a2i_h/a2i_s qrels referenced prefixed query-ids while the
    queries split used unprefixed ids, leaving 0/120 retrievable queries; rebuilt
    and pushed corrected dataset to mteb/coco-modality-equivalence.
  • Add generated descriptive stats for all six task directions.
  • Reorder all entries in tasks/retrieval/eng/init.py (ruff --fix).
  • Point analysis scripts at fixed dataset and rerun results
  • Update _REPO in directional_asymmetry.py, modality_gap.py, and
    sampling_budget.py from rakshi719/coco-modality-equivalence to the
    corrected mteb/coco-modality-equivalence.
  • Rerun all four analysis scripts (directional_asymmetry, modality_gap,
    sampling_budget, latent_dimensions) and refresh their results/*.json.

Co-authored-by: Rakshitha Ireddi <rakshithaireddi@Rakshithas-MacBook-Pro.local>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (1dfd69a)

  • Create Ops MM models (2B, 7B) (#5332)

  • Create ops_mm_models

Co-Authored-By: Deep Shah <21212684+deep9539@users.noreply.github.com>

  • Update ops_mm_models.py

  • lint

Co-authored-by: Deep Shah <21212684+deep9539@users.noreply.github.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (4e24e0c)

  • Add WeMM embedding model. (#5333)

  • Add WEMM models

  • Update pyproject.toml

  • Use InstructSentenceTransformerModel

  • fix video and better support instruction prompt

  • Update wemm_models.py

  • Update wemm_models.py

  • reformat and fix batch padding

  • fix padding

  • fix padding

  • Update wemm to use sentence_transformer

  • Add init method

  • Reformat to pass linter

  • Add comment to explain the permute

  • address comments

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (951d86b)

  • Add RzenEmbed model (#5339)

  • first commit

  • handle empty task instruction

  • resolve naming discrepancy

  • fix things

  • add tensor support

  • dimension handling

  • fix error

  • visual embeddings to language model dimension

  • Update rzen_embed_model.py

Fix system prompt, image + video coprocessing.

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

  • delete not required files.

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

  • resolve comments

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

  • fix error

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

  • update image embed dimension

Co-Authored-By: nehal309 <20415842+nehal309@users.noreply.github.com>

  • Add change to make rzen embed compatible to >= 4.57.0

  • add rzen training data and embedding_params

  • lint

  • Update rzen_embed_model.py

Co-authored-by: nehal309 <20415842+nehal309@users.noreply.github.com>
Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (74ee71c)

  • dataset: add ToolRetrieval benchmark task (#5410)

  • feat(retrieval): add ToolRetrieval benchmark task (#3628)

  • fix(retrieval): reproduce published ToolRet results

ToolRetrieval scored ~11 NDCG@10 above the paper because it differed from the
reference implementation in three ways, each making retrieval easier:

  • queries retrieved from their own category's tools rather than the pooled
    44,453-tool corpus, which is what eval_retrieval(category=&#34;all&#34;) does
    upstream and what the published tables report
  • subset scores micro-averaged over queries instead of taking the unweighted
    mean over the 35 retrieval tasks; this dominates web, whose tasks range
    from 11 to 1,100 queries
  • instructions joined as "{query} {instruction}" rather than the reference
    template "Instruct: {instruction}\nQuery: {query}"

Pool the corpus, expose the 35 retrieval tasks as subsets so MTEB's
cross-subset mean matches the paper's aggregation, and add
ToolRetrievalInstruction for the w/ inst. setting (Table 5) alongside
ToolRetrieval (w/o inst., Table 4).

all-MiniLM-L6-v2 w/ inst., NDCG@10, paper Table 5 in parentheses:
web 13.23 (12.77), code 32.38 (31.59), customized 33.29 (32.24). Mean absolute
delta drops from 10.91 to 0.76. Reproduce with scripts/reproduce_toolret.py,
which also raises max_seq_length to 512 as the reference implementation does;
this model's card ships 256 and web swings ~5 NDCG@10 on that alone.

  • feat(retrieval): reproduce ToolRet paper trends in the example script

The example script now carries the reference implementation's per-model
handling, without which the published numbers do not come out: SentenceTransformer
vs fp16 AutoModel dispatch, per-family pooling, the L2-normalization skip for
contriever and gtr-t5, per-family prompt templates, min(max_position_embeddings,
2048), and word-level truncation. It takes --model/--all/--settings and prints a
PASS/FAIL against the published values, defaulting to Table 5.

Over 9 baselines the paper's findings replicate:

  • instructions help every model, mean gain +11.26 NDCG@10 vs the paper's +11.34
    (Pearson 0.958 across models)
  • the model ranking replicates (Pearson 0.988, Spearman 0.967)
  • subset difficulty orders the same way, web < code < customized
  • 85% of metrics land within 2.0 NDCG@10, mean absolute delta 1.39; 92% for
    w/ inst. excluding Tool-COLT, whose public checkpoint appears not to be the
    one the authors evaluated

Exact per-cell agreement is not available: the reference print_results() computes
a size-weighted (micro) mean while the published tables match an unweighted
(macro) one, so the paper's numbers were not produced by the released code.

  • fix(scripts): correct batching and refuse unsafe model fallbacks

Two bugs in the reproduction script, both found by running it over every
baseline rather than one:

  • the token-budget batch size was computed from the shortest text in a sorted
    batch, so short queries produced enormous forward passes and tripped a
    device-side assert; it is now capped at MAX_BATCH, and the SentenceTransformer
    path sizes its batch from max_seq_length instead of a hardcoded 32, which was
    exhausting a 24GB card at 2048 tokens
  • a trust_remote_code=False fallback was added for checkpoints whose remote code
    lags transformers, but for gte-Qwen2 that loads a different model: its custom
    code makes attention bidirectional where the native architecture is causal, and
    it scored ~19 NDCG@10 below the published value while looking plausible. The
    fallback is removed so the load fails loudly instead.

Coverage is now 10 of Table 5's 26 rows -- every single-vector dense retriever
under 7B that runs in this harness. gtr-t5-large is included despite scoring
below gtr-t5-base, inverting the paper's ordering for that pair; it is reported
as measured rather than dropped, and the docstring records both the headline
figures and the figures excluding it.

  • fix(scripts): use native MTEB ToolRetrieval evaluation

  • chore: remove Modal evaluation helper

  • reupload

  • remove unnecessary script

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a768963)

  • Add ModelMeta for NGA-KR/ko-embed-cls (#5446)

  • Add ModelMeta for NGA-KR/ko-embed-cls

  • Fix ModelMeta for NGA-KR/ko-embed-cls (lint, model_type, prompts)

  • Add n_embedding_parameters

Co-authored-by: Isaac Chung <isaac.chung@foam.io> (cb65aad)

  • task: add BioVITA multimodal retrieval (#5153)

  • task: add BioVITA multimodal retrieval

  • fix: use fixed-format datasets for BioVITA

  • fix: simplify BioVITA task metadata and loading

  • fix: classify BioVITA tasks as reranking

  • fix: address BioVITA reranking review

  • style: format BioVITA data script

  • fix: return BioVITA audio fallbacks

  • fix: add BioVITA descriptive statistics

  • test: update BioVITA reranking quality allowlist

  • docs: clarify BioVITA tie handling

  • refactor: remove _BioVITAReranking base class, add task_specific_scores to each class explicitly

Also fix BibTeX formatting in webvid_covr files to satisfy pre-commit hook.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

Co-authored-by: Michelle Yang <myang333@users.noreply.github.com>
Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com> (6ee088e)

  • Add ModelMeta for NGA-KR/ko-embed-nli (#5447)

  • Add ModelMeta for NGA-KR/ko-embed-nli

  • Add n_embedding_parameters (8b85f40)

  • add Nemotron model citations and public training references (#5445)

  • fix: add Nemotron citations and Embed-VL training code link

  • fix: link Nemotron 3 public training datasets

  • fix: link the full Nemotron 3 training data sections (4fe4737)

  • add OpenMDW license metadata for corresponding models (#5444)

fix: update OpenMDW license metadata for Nemotron models (5826242)

  • recognize OpenMDW-1.1 as an open license (#5443)

fix: recognize OpenMDW-1.1 as an open license (1183517)

  • Add ModelMeta for NGA-KR/ko-embed-v0 (#5440)

  • Add ModelMeta for NGA-KR/ko-embed-v0

  • lint

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (a8bce9d)

Contributors

Samoed
Assets 6
Loading