Simple cmd-line league system for bot contests.
Install the latest version with pip install psyleague --upgrade
AFAIK, it requires python 3.8 or newer.
You can see the latest changes in the changelog.txt.
Note: if you encounter any bugs/problems, feel free to contact me on twitter/discord and tell me about the issue.
- League with simple automatic matchmaking system: start the league server, add bots, look at the results table
- Add/remove/modify bots without restarting the server
- All data is stored in a human-readable format
- Easy way of adding metadata to your games and the display those stats on the leaderboard; it also allows for rating recalculation on a specific subset of games
- Should work with all programming languages and all possible platforms
- [Coming in 0.5.0?] Support for different ranking models
- Install
psyleague
(viapip install psyleague --upgrade
) - Create a play_game script that is invoked every time psyleague wants to run a new game. This is usually either a wrapper on a provided referee or your own game simulator. Please see this example for explanation about the JSON format.
- Run
psyleague config
in your contest directory to create a new config file - In
psyleague.cfg
you have to modifycmd_bot_setup
andcmd_play_game
.cmd_bot_setup
is executed immediately when you add a new bot.cmd_play_game
is executed whenpsyleague
wants to play a single game. %DIR% ->dir_bots
(from config), %NAME% ->BOT_NAME
, %SRC% ->SOURCE
(orBOT_NAME
ifSOURCE
was not provided), %P1% & %P2% ->BOT_NAME
of the player 1 & player 2 bots. - If your game has more than 2 players, you should change
n_players
in the config. - Run
psyleague run
in a terminal - this is the "server" part that automatically plays games. In order to kill it, use keyboard interrupt (Ctrl+C). - In a different terminal, start adding bots by running
psyleague bot add BOT_NAME -s SOURCE
to add a new bot to the league. As soon as you have 2 bots added,psyleague run
will start playing games. - Run
psyleague show
to see the current leaderboard - Remember to update
n_workers
(in the config or viarun --workers n_workers
) in order to play more games simultaneously
Your play_game script should print a valid JSON that contains 4 fields. P refers to the number of players.
ranks
: a list of length P, that describes the final placements of every player; it follows the same format as trueskill libraryerrors
: a list of length P that describes if there was an error (timeout, crash, etc.) for a particular playertest_data
: an object (dictionary) that contains metadata related to this particular test/seed.player_data
: a list of length P that contains metadata related to each of the players.
The following python code should work with most of the CodinGame contests assuming referee.jar
contains the referee. If your referee works with brutaltester it also works with psyleague.
import sys, subprocess, random, json, tempfile, os
if __name__ == '__main__':
f, log_file = tempfile.mkstemp(prefix='log_')
os.close(f)
seed = random.randrange(0, 2**31)
n_players = len(sys.argv) - 1
cmd = 'java -jar referee.jar' + ''.join([f' -p{i} "{sys.argv[i]}"' for i in range(1, n_players+1)]) + f' -d seed={seed} -l "{log_file}"'
task = subprocess.run(cmd, shell=True, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
with open(log_file, 'r') as f:
json_log = json.load(f)
os.remove(log_file)
p_scores = [int(json_log['scores'][str(i)]) for i in range(n_players)]
rv = {}
rv['ranks'] = [sum([int(p_score < p2_score) for p2_score in p_scores]) for p_score in p_scores] # assumes higher score is better
rv['errors'] = [int(p_score < 0) for p_score in p_scores] # assumes negative score means error
rv['test_data'] = {'seed': seed}
rv['player_data'] = [{} for _ in range(n_players)]
for player, key in enumerate([str(i) for i in range(n_players)]):
for data in json_log['errors'][key]:
if not data: continue
for line in [line.strip() for line in data.split('\n')]:
vs = line.split(' ')
if len(vs) < 4: continue
if vs[0] == '[TDATA]': rv['test_data'][vs[1]] = vs[3]
if vs[0] == '[PDATA]': rv['player_data'][player][vs[1]] = vs[3]
if vs[0] == '[PDATA+]': rv['player_data'][player][vs[1]] = rv['player_data'][player].get(vs[1], 0.0) + float(vs[3])
print(json.dumps(rv))
In order to add metadata to test_data, just print [TDATA] key = value
to stderr in your bot. Similarly, print [PDATA] key = value
for adding metadata to player_data. Use [PDATA+] key = value
if you want to store the sum of all of the values instead only the last one.
- Most of the potential errors come from either having incorrect referee or a mistake in your play_game script. It's recommended to run your
cmd_play_game
manually and see if it correctly prints out JSON string to stdout. - Set
n_workers
to 1 and run server with verbose turned on:psyleague run --verbose
to see if yourcmd_play_game
script is called correctly.
- trueskill
- this is the same model that CodinGame uses, except here you have the ability to reduce
tau
so that the ranking can stabilize after a while. Unless you're running 10K+ games per bot, there's probably no reason to reducetau
even more.
- this is the same model that CodinGame uses, except here you have the ability to reduce
Matchmaking model is very simple:
- If there are any bots with less than
mm_min_matches
games and we have "rolled" belowmm_min_matches_preference
: play a game between a bot with not enough games and a random bot - Otherwise: play a game between two random bots
TBD
psyleague
is not going to be backward compatible. Every new version might break the format of any of the files and/or config.- Most of the referees for CodinGame detect if your bot goes above allowed time for each turn. If you spawn too many workers your bots will start timing out randomly.
- Config file is read only once at the startup. If you have updated config file, you have to restart
psyleague run
to reflect the changes - You can modify
psyleague.db
to make direct changes to the bots/stats, but don't do that whilepsyleague run
is running. In order to reset everything, it's enough to deletepsyleague.db
&psyleague.games
. - If you want to see the list of planned changes, see the top of the source file