Skip to content

2.6.0

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 30 Dec 12:02
· 1150 commits to main since this release
2.6.0
1c873b4

2.6.0 (2025-12-30)

Feature

  • feat: Add leaderboard CLI command (#3802)

  • feat: add leaderboard CLI command with cache-path option

  • test: add comprehensive tests for leaderboard CLI command

  • try to fix install

  • fix: lazy-load leaderboard to avoid requiring deps for CLI

  • Update mteb/cli/build_cli.py

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>

  • make lint

  • remove AGENTS.md

  • move import to top of file

  • log the default cache path

  • Improve leaderboard tests to verify actual cache paths

Address PR feedback by modifying leaderboard tests to verify the actual
cache paths passed to get_leaderboard_app instead of mocking ResultCache.

  • Updated test_leaderboard_custom_cache_path to create real ResultCache instances
    and verify the correct custom cache path is used
  • Updated test_leaderboard_default_cache to verify the default cache path is used
  • Removed ResultCache mocking in favor of testing actual cache behavior
  • Used patch.dict to mock the leaderboard module import while preserving
    real cache functionality

This provides better test coverage by validating that the cache objects
passed to the leaderboard app have the correct paths, as suggested in
PR comment: #3802 (comment)

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Combine leaderboard cache tests using pytest parametrize

Address PR feedback by combining test_leaderboard_custom_cache_path and
test_leaderboard_default_cache into a single parametrized test.

  • Created test_leaderboard_cache_paths with parametrize decorator
  • Tests both custom cache path and default cache path scenarios
  • Each test case covers different host, port, and share configurations
  • Removed redundant test_leaderboard_args as functionality is now covered
    by the parametrized test
  • Improved test maintainability by reducing code duplication

This addresses PR comment: #3802 (comment)
"Can be combined with the following test using a parametrize argument"

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Update make run-leaderboard to use new CLI and remove app.py main block

Address PR feedback by updating the project to use the new leaderboard CLI:

  • Updated Makefile run-leaderboard target to use python -m mteb leaderboard
    instead of python -m mteb.leaderboard.app
  • Removed the if __name__ == &#34;__main__&#34;: block from mteb/leaderboard/app.py
    as this functionality is now handled by the CLI command

This completes the integration of the new leaderboard CLI command into
the project's build system and removes deprecated direct module execution.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • feat: add theme and head parameters to leaderboard CLI

  • fix: suppress leaderboard warnings on CLI launch

  • test: update leaderboard tests for theme and head params

  • Revert "Update make run-leaderboard to use new CLI and remove app.py main block"

This reverts commit d4df501.

  • Update mteb/cli/build_cli.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

  • docs: update leaderboard CLI usage

  • update docs to show defaults

  • fix: apply ruff formatting

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (1a64ed6)

Assets 6
Loading