SWA attempt

[sethtroisi]

I'm trying out SWA

Write-up: glinscott/leela-chess#159

Original post: leela-zero/leela-zero#814

Code: https://github.com/sethtroisi/minigo/tree/swa?files=1

https://github.com/sethtroisi/minigo/blob/swa/swa.py

I have ringmaster test running right now between swa-2, swa-4, swa-8, swa-10, and 000303-olympus

[sethtroisi]

minigo-cc-swa-2 v minigo-cc-swa-4 (100/100 games)
board size: 19   komi: 7.5
                  wins              black         white       avg cpu
minigo-cc-swa-2     39 39.00%       16 32.00%     23 46.00%    111.23
minigo-cc-swa-4     61 61.00%       27 54.00%     34 68.00%    111.48
                                    43 43.00%     57 57.00%

minigo-cc-swa-4 v minigo-cc-swa-8 (100/100 games)
board size: 19   komi: 7.5
                  wins              black         white       avg cpu
minigo-cc-swa-4     52 52.00%       25 50.00%     27 54.00%    114.76
minigo-cc-swa-8     48 48.00%       23 46.00%     25 50.00%    114.81
                                    48 48.00%     52 52.00%

minigo-cc-swa-8 v minigo-cc-swa-10 (100/100 games)
board size: 19   komi: 7.5
                   wins              black         white       avg cpu
minigo-cc-swa-8      52 52.00%       26 52.00%     26 52.00%    116.53
minigo-cc-swa-10     48 48.00%       24 48.00%     24 48.00%    116.51
                                     50 50.00%     50 50.00%

minigo-cc-swa-2 v minigo-cc-303 (100/100 games)
board size: 19   komi: 7.5
                  wins              black         white       avg cpu
minigo-cc-swa-2     47 47.00%       29 58.00%     18 36.00%    112.41
minigo-cc-303       53 53.00%       32 64.00%     21 42.00%    112.34
                                    61 61.00%     39 39.00%

minigo-cc-swa-10 v minigo-cc-303 (100/100 games)
board size: 19   komi: 7.5
                   wins              black         white       avg cpu
minigo-cc-swa-10     61 61.00%       25 50.00%     36 72.00%    111.15
minigo-cc-303        39 39.00%       14 28.00%     25 50.00%    110.77
                                     39 39.00%     61 61.00%

[sethtroisi]

Per LZs implementation it seems like we should run another couple of batches of training so that batch norms can be recalculated

leela-zero/leela-zero@94f48bb#diff-533b2cfc6eb41b6d5b8954e445851082R453