Add the arena leaderboard - #12
Merged
Merged
Conversation
Ranks agents fairly: every competitor plays both the werewolves (against a fixed reference village) and the village (against fixed reference werewolves) under identical seeds. Score is the mean of the two win rates. - new deepwolf/arena/leaderboard.py: Leaderboard, LeaderboardReport, LeaderboardEntry; reuses the existing Arena for each match-up. - new `deepwolf leaderboard` CLI command with a --markdown export. - 7 new tests. Closes #4 Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Owner
Author
|
Code review (post-merge record).
Nice — this is what turns the arena into an actual benchmark. Merged via squash. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Implements the arena leaderboard requested in #4.
A leaderboard ranks agents fairly. Each competitor is measured under
identical, seeded conditions:
Its score is the mean of the two win rates. Because both match-ups reuse the
same seeds for every competitor, the ranking is reproducible and the only
variable is the agent itself.
Closes #4
Changes
deepwolf/arena/leaderboard.py—Leaderboard,LeaderboardReport(with
render()andto_markdown()),LeaderboardEntry. It reuses theexisting
Arenafor each match-up rather than duplicating the game loop.deepwolf leaderboardCLI command, with--markdown PATHto export apaste-ready table and
--provider env/--modelto add a real model as acompetitor.
Checklist
ruff check .passesmypy deepwolfpassespytestpasses (50 tests; 7 new)CHANGELOG.mdandREADME.mdupdatedNotes for reviewers
A "competitor" is a faction-agnostic
AgentBuilder(Callable[[int], Agent]),so the same agent can be slotted into either side.
Leaderboard._arenacomposestwo builders into the
(player_id, role) -> Agentfactory theArenaexpects.