Pool of matches

Introduction

When strategy selection methods face each other, they select StarCraft strategies that battle each other in a match. But, to speed things up, we draw the result of this match from a pool of previously executed matches, instead of actually executing StarCraft.

The pool of matches is a .txt file generated by StarCraft AI Tournament manager software (link to original project and fork with persistent knowledge disabled).

Pool of matches file

Currently, StarcraftNash's pool of matches file is generated by StarCraft AI Tournament manager software as html/detailed_results.txt (when the software is told to generate detailed match results). Each line of the file refers to a match. There's a lot of information about each match, but for StarcraftNash, only the winner and loser bot are of interest. They are the third and fourth columns of the file, respectively.

For the sake of completeness, see below this file's line format. Consider the following abbreviations: W: winner, L: loser, sc: score, to: timeouts. Also, duration is expressed in in-game timesteps.

  match_id     round_id     winner     loser     crash     timeout     map     duration     W_sc     L_sc     duration     W_55ms-to     W_1000ms-to     W_10000ms-to     L_55ms-to     L_1000ms-to     L_10000ms-to     W_Addr     L_Addr     Start-timestamp     Finish-timestamp

A typical line looks like the following:

  0     0     UAlbertaBot            Aiur               -               -           (4)Fortress.scx    21330    39622    22111       3       0       0       0       0       0  /192.168.56.101  /192.168.56.102  20160529_075134  20160529_075313

In the example above, UAlbertaBot has beaten Aiur in map Fortress.

How the pool is read

When two strategy selection techniques face each other and select which strategy/bot will play a match, StarcraftNash looks into the pool of matches and returns the result of the first match between selected strategies/bots. If the same strategies/bots are selected again, the next match between them is looked up in the pool and its result is returned. StarcraftNash proceeds this way until all matches are looked up. If the same bots are selected again, then StarcraftNash prints a warning and starts over from the first match in the pool. For example, consider the simplified pool below, showing only the winner and loser of a match.

A B
C D
B A
D C
A B

If C and D are selected by some strategy selection techniques, StarcraftNash will look up at the second line of the pool and return C as the winner. If they're selected again, StarcraftNash will look up at the fourth line and return D as the winner. If they're selected yet again, StarcraftNash issues a warning and returns the result on the second line again.

This procedure allows to capture the fact that a bot can adapt to another and change its relative performance, although this can mess up with StarcraftNash's experiment results for now. See the discussion below for more information.

Generating the pool of matches

To generate the pool of matches file, you need to run a StarCraft AI tournament. You can do it with persistent knowledge either enabled or disabled. By default, tournament manager software runs with persistent knowledge enabled. When persistent knowledge is enabled, bots write data after a match in a directory which is copied to a 'read' directory after a tournament round finishes. In next round, the bot is able to read the file and use the knowledge it gathered previously. This enables bots to learn about their opponents and improve performance by adapting to them.

While persistent knowledge is desirable for tournaments, it can interfere in StarcraftNash's results. This happens because some strategy selection methods assume that victory probabilities among strategies/bots are stationary, that is, it does not change as the strategies/bots play repeatedly. Suppose bot A consistently loses to bot B at first, but starts winning by adapting to it after a number of matches. Currently, StarcraftNash's strategy selection methods are not able to handle to this situation, because they receive the victory probabilities as prior information which is not updated. Handling non-stationary victory probabilities among strategies is future work.

An advantage of disabling persistent knowledge is that repeated tournaments can be executed just by shuffling the pool of matches between repetitions. Without persistent knowledge, victory probabilities between bots are expected to remain the same along different portions of the pool whereas with persistent knowledge it can change. Therefore, shuffling a pool generated WITHOUT persistent knowledge is harmless whereas shuffling a pool generated WITH persistent knowledge will mix up results of bots under different knowledge conditions.

In order to run a StarCraft tournament with persistent knowledge disabled, you must use the manager software from the forked project and change the AllowRead flag in server_settings.ini to no.

Below you can find the server_settings.ini in a tournament between Aiur, CruzBot, NUSBot, Skynet and Xelnaga (AIIDE 2015 version of the bots was used) in Fortress map. A 1000-round tournament using this file was used to generate results_demo/fortress1000.txt (which is here).

Bot  Aiur                 Protoss   dll  BWAPI_374 
Bot  CruzBot              Protoss   dll  BWAPI_374 
Bot  NUSBot               Protoss   dll  BWAPI_374 
Bot  Skynet               Protoss   dll  BWAPI_374  
Bot  Xelnaga              Protoss   dll  BWAPI_374 

Map maps/aiide/(4)Fortress.scx

GamesListFile games.txt
ResultsFile results.txt
DetailedResults yes
ServerPort 1337
ClearResults ask
ResumeTournament ask
AllowRead no

TMLocalSpeed 0
TMFrameSkip 256
TMGameFrameLimit 85714

TMTimeout 55 320
TMTimeout 1000 10
TMTimeout 10000 1

TMDrawBotNames         true
TMDrawTournamentInfo   true
TMDrawUnitInfo         true

TournamentType          AllVsAll