Check fairness in the Dojo

[kristjan]

@tcollier

This script sets up two truly random bots and reveals that the Dojo biases towards the first one passed in, which somehow wins 45% of the time instead of 33%. As proof, it reverses the order and gets the same result.

$ bin/check_fairness 10000
### Running competitors in order (Devil 0, Devil 1)
Devil 0 |   4530    |   45.3%
Devil 1 |   3301    |   33.0%
draw    |   2169    |   21.7%

### Reversing competitor order (Devil 1, Devil 0)
Devil 0 |   3344    |   33.4%
Devil 1 |   4497    |   45.0%
draw    |   2159    |   21.6%

[kristjan]

Here's a broken bout. One ignores the seed and runs truly randomly, while the Spaz uses the seed. Something goes horribly wrong in Match 2, where One should lose 10 to 4.

Starting match with seed: 0.6790859699249268
  One       |  SpazmanianDevil
------------+-------------------
 *paper     |  rock
  rock      | *paper
 *paper     |  rock
  paper     |  paper
 *scissors  |  paper
  paper     | *scissors
 *paper     |  rock
 *rock      |  scissors
 *paper     |  rock
  rock      |  rock
 *paper     |  rock
  scissors  |  scissors
 *scissors  |  paper
  paper     | *scissors
 *rock      |  scissors
  scissors  | *rock
  rock      | *paper
 *scissors  |  paper
  rock      | *paper
  rock      | *paper
  rock      |  rock
 *scissors  |  paper
 *paper     |  rock
 *paper     |  rock
  rock      | *paper

One won match 1

Starting match with seed: 0.6824829578399658
  One       |  SpazmanianDevil
------------+-------------------
  rock      |  rock
 *scissors  |  paper
 *paper     |  rock
  paper     |  paper
  rock      | *paper
  paper     | *scissors
  rock      |  rock
  paper     | *scissors
  rock      |  rock
  scissors  | *rock
  scissors  | *rock
  paper     | *scissors
  rock      | *paper
  scissors  |  scissors
  scissors  |  scissors
 *paper     |  rock
 *scissors  |  paper
  paper     |  paper
  paper     |  paper
  paper     |  paper
  scissors  | *rock
  paper     |  paper
  scissors  | *rock
  scissors  | *rock
  paper     |  paper

Match 2 was a draw

Starting match with seed: 0.6851890087127686
  One       |  SpazmanianDevil
------------+-------------------
  rock      |  rock
 *scissors  |  paper
  rock      |  rock
 *scissors  |  paper
  paper     |  paper
  paper     | *scissors
  rock      |  rock
  scissors  |  scissors
  scissors  | *rock
  rock      |  rock
  scissors  | *rock
  scissors  |  scissors
  paper     |  paper
 *rock      |  scissors
 *rock      |  scissors
 *paper     |  rock
 *scissors  |  paper
  rock      | *paper
 *scissors  |  paper
  paper     |  paper
  rock      |  rock
  rock      | *paper
  rock      |  rock
  scissors  | *rock
  rock      | *paper

Match 3 was a draw

One beat SpazmanianDevil!!!

[tcollier]

Wow, great find! I wrote this code over 3 years ago so I will have to refamiliarize myself with it to dig into it.

LGTM