Skip to content

Generate one rank per evaluation cohort - #8

Merged
nipunbatra merged 1 commit into
mainfrom
agent/global-leaderboard-rank
Jul 17, 2026
Merged

Generate one rank per evaluation cohort#8
nipunbatra merged 1 commit into
mainfrom
agent/global-leaderboard-rank

Conversation

@nipunbatra

Copy link
Copy Markdown
Member

What changed

  • generate rank and a dedicated evaluation-cohort digest in the Python leaderboard builder
  • keep sequence length, epochs, tuning, container, and runtime provenance in the full comparison record without letting them reset public rank
  • render the generated rank directly in the site
  • regenerate the checked-in JSON and CSV, yielding one cohort with ranks 1–17

Root cause

The browser grouped rows by the full comparison-protocol digest. That digest intentionally changes for model context length and runtime provenance, so DAE/BERT and Mean each started a new rank group despite using the same REDD/fridge evaluation cohort.

Verification

  • uv run ruff check src tests
  • uv run pytest — 129 passed, 1 skipped
  • regenerated leaderboard JSON/CSV are byte-for-byte reproducible
  • explicit regression coverage for differing sequence lengths and container revisions

@nipunbatra
nipunbatra merged commit 46451e1 into main Jul 17, 2026
1 check passed
@nipunbatra
nipunbatra deleted the agent/global-leaderboard-rank branch July 17, 2026 12:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant