Skip to content

2.6.1

Choose a tag to compare

View all tags
@KennethEnevoldsen KennethEnevoldsen released this 30 Dec 21:57
· 1144 commits to main since this release
2.6.1
b895af5

2.6.1 (2025-12-30)

Fix

  • fix: Download cached results zip from cached-data branch (#3795)

  • Optimize leaderboard startup by downloading cached results from cached-data branch

  • Modify _load_results() to first try downloading __cached_results.json.gz from the cached-data branch
  • Only fallback to full repository clone if the direct download fails
  • Add gzip decompression to handle the compressed cache file
  • This reduces startup time significantly by avoiding full repo cloning when possible
  • Added comprehensive logging to track download progress and fallback behavior

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • make lint

  • Fix leaderboard stability test with enhanced debugging

  • Remove prevent_thread_lock=True to keep Gradio process alive
  • Add comprehensive exception handling for HTTP, gzip, and file operations
  • Optimize test completion with HTTP 200 health checking (300s → ~140s)
  • Add detailed logging and warning suppressions for better debugging

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Update tests/test_leaderboard.py

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

  • Add comprehensive tests for leaderboard caching exception handling
  • Add 46 unit tests covering HTTP downloads, gzip decompression, file I/O, and JSON validation
  • Reorganize leaderboard tests into focused modules for better maintainability
  • Update Makefile with improved leaderboard test commands

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Increase cached results download size limit to 500MB

The cached results file has grown to ~92.7MB, exceeding the previous 50MB limit.
This change increases the limit to 500MB to accommodate current and future file sizes.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Fix leaderboard tests by adding missing dependency to install-for-tests

GitHub Actions were failing because cachetools was not installed during CI test runs.
The leaderboard extra was already defined with cachetools>=5.2.0 but wasn't included
in the install-for-tests target used by CI.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Remove LogFlusher functionality from leaderboard app

Addresses PR comment feedback indicating the log flushing optimization
was unnecessary at this stage. Removes:

  • LogFlusher class with batching logic
  • Global _log_flusher instance
  • _flush_logs() wrapper function
  • All calls to _flush_logs() throughout the app
  • Complete test file test_log_flushing.py

Leaderboard functionality remains unchanged and tests pass.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • remove _validate_benchmark_json

  • Refactor leaderboard caching to use ResultCache and consolidate tests

Move download_cached_results_from_branch to ResultCache class and reduce TestDownloadCachedResultsFromBranch from 23 to 13 test cases while maintaining full coverage.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • Apply suggestions from code review

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

  • lint and remove unreachable code

  • Move shared test fixtures to parent conftest.py

  • Created tests/conftest.py with shared fixtures (mock_benchmark_json,
    mock_invalid_json, mock_gzipped_content) for use across all tests
  • Removed duplicate fixtures from tests/test_leaderboard/conftest.py
  • Kept leaderboard-specific fixtures in test_leaderboard/conftest.py
  • Fixes TestDownloadCachedResultsFromBranch test failures by making
    fixtures accessible to test_result_cache.py

All 25 tests now passing (23 in test_result_cache.py, 2 in test_integration.py)

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

  • make method private

  • Fix content type validation test to match implementation behavior

The test_content_type_handling test was expecting warnings for unexpected
content types, but the actual implementation raises exceptions. Updated test
to use pytest.raises() for proper exception validation.

🤖 Generated with Claude Code

Co-Authored-By: Claude Sonnet 4 <noreply@anthropic.com>

  • update cache based on review comments

  • type check

  • Remove unused leaderboard_test_config fixture

  • fix: remove unused mock_invalid_json fixture

  • rm AGENTS/,d

  • reduce number of excepts in app.py

Co-authored-by: Claude Sonnet 4 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com> (3303087)

Unknown

  • add model: bflhc/Octen-Embedding-4B (#3816) (6dcbf9f)

  • Add filter for model type (#3799)

  • Add filter for model type

  • fix literal issue

  • fix

  • remove white space

  • remove logic in filter_tasks

  • remove info in leaderboard

  • add tests

  • update tests

  • add default in model types

  • fix model filter

Co-authored-by: Roman Solomatin <36135455+Samoed@users.noreply.github.com> (6f4627e)

Assets 6
Loading