Skip to content

2.8.1

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 17 Feb 23:21
· 992 commits to main since this release
2.8.1
c436cbb

2.8.1 (2026-02-17)

Fix

  • fix: Remove duplicate citations and add test to prevent it going forward (#4032)

  • test: add test to detect duplicate citations

  • quality

  • move changes to task file

  • fix falsepositives

  • update citations

  • add models and benchmarks

  • search close titles

  • fix maeb

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (377f1b4)

Unknown

  • benchmark: add 6 new VisRAG retrieval tasks and corresponding stats (#4059)

  • dataset: add 6 new VisRAG retrieval tasks and corresponding stats

  • Introduced VisRAGRetArxivQA, VisRAGRetChartQA, VisRAGRetInfoVQA, VisRAGRetMPDocVQA, VisRAGRetPlotQA, and VisRAGRetSlideVQA classes for various retrieval tasks.
  • Added JSON files containing descriptive statistics for each task, including sample counts, image dimensions, and query statistics.
  • Updated the retrieval module's init.py to include the new tasks in the module exports.

  • fix a linter error

  • dataset: introduce VisRAG Retrieval Benchmark

  • fix: metadata update of VisRag

  • Update benchmark metadata

  • Update VisRAG datasets metadata, including one-line description and the domains

  • Update slideVQA domain

  • Add Aliases for VisRAG

  • Fix bibtex format

  • Update dataset metadata to point to the mteb versions

  • Remove redundant data loading

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (7a9e653)

  • fix: Remove MAEB+ and MAEB(extended) from leaderboard and add "beta" to all MAEB (#4103)

  • fix: Remove MAEB+ and MAEB(extended)

related to #3470

We could consider keeping the two benchmarks (would still need to be beta as paper is review) so they could change.

  • fix: Remove MAEB+ and MAEB(extended) from leaderboard and make add "beta" to all MAEB

related to #3470

  • Remove MAEB+ and MAEB(extended) from leaderboard
  • Added the beta tag to denote that these might change

Currently implemented it as keeping the two temporary benchmarks. We could consider removing them as well (I am unsure how much of a burden it is for us to maintain them), but I would probably not add them to the leaderboard.

All of these changes should be backward compatible

  • docs: Added whatsnew

  • implement fixes

  • update description to explain beta status (77ac52b)

Assets 6
Loading