Script for converting minigo weights

[Ttl]

Converts minigo weights to work with LZ. Some minigo weights are available here: https://console.cloud.google.com/storage/browser/minigo-pub

Some tests:

LZ weights 147 vs. LZ with minigo weights 303. 1000 visits, minigo weights has 3 handicap stones:

lz_minigo_303 v lz_147 (15/100 games)
board size: 19   handicap: 3 (free)   komi: 7.5
                wins                  avg cpu
lz_minigo_303      4 26.67%   (black)  945.66
lz_147            11 73.33%   (white)  615.45

Minigo vs LZ using the same 303-olympus weights with 100 playouts. LZ arguments: -g -d -t 1 --noponder -p 100 --timemanage off -r 5. Minigo used the default settings.

lz_303 v minigo_303 (100/100 games)
board size: 19   komi: 7.5
             wins              black         white       avg cpu
lz_303         86 86.00%       42 84.00%     44 88.00%    107.65
minigo_303     14 14.00%       6  12.00%     8  16.00%   4449.02
                               48 48.00%     52 52.00%

[amj]

was this vs minigo's python code or the c++ code?

It obviously seems a little off that the same weights should be so bad in a different engine and it makes me wonder if we have a bug. Can you show me the command used to invoke minigo?

[Ttl]

Python code. Command was BOARD_SIZE=19 python3 main.py gtp -l v7-19x19_models_000303-olympus -v 3.

At least some of the strength difference is caused by batching and lack of FPU reduction.

Also I noticed that minigo has lot of batch norm gamma values that are zero causing the whole output plane to be zero. For example 65 planes of the input convolution have zero gamma and two next convolutions have 12 and 25 zeroed planes. This seems little extreme to me.

[brilee]

To summarize what I think are possible differences in MCTS fiddly bits, there's:

• parallelism in tree search; up to how many virtual losses are allowed before waiting for GPU to return batched board evaluations?
• cPUCT params
• Q initialization (FPU stuff)
• anything else?

[brilee]

@amj was saying that LZ might be working with parallel reads = 1 whereas we're using parallel reads=8 by default. If that's correct, would you be open to trying the matchup again with --parallel_readouts=1?

[Ttl]

I just wanted to verify that the weights were converted correctly. The match was run on a laptop and even with just 100 playouts it took over a day to play 100 games. I have other uses for it right now and don't want to repeat the match.

[amj]

Ok, i can definitely understand that -- if you can compile our cc version it should be much faster :)

Edit-to-add: the disparity in perf numbers makes me wonder if we had GPU enabled for minigo?

[brilee]

Okay - thank you very much for reporting the results here.

[Ttl]

The laptop this was played on had only Intel integrated GPU. As far as I know tensorflow can't use it, but LZs OpenCL code supports it. That's why there is such a large difference in the CPU use.

[sethtroisi]

I have everything running on my computer (which has a GPU)

Control File

competition_type = 'playoff'
description = """ Testing MiniGo models in leelaz src code """

record_games = True
stderr_to_log = True

players = {
    'lz-minigo-303' : Player("./leelaz -g -d -t 1 --noponder -p 100 --timemanage off -r 5 -w 000303-olympus_converted.txt"),
    'minigo-303' : Player(
        "python3 main.py gtp -l models/000303-olympus -v 3 --num_readouts 100",
        cwd="~/Projects/minigo",
        environ={'BOARD_SIZE' : '19'},
    ),
}

board_size = 19
komi = 7.5

matchups = [
    Matchup('lz-minigo-303', 'minigo-303', alternating=True, scorer='players', number_of_games=20),
]

Results

lz-minigo-303 v minigo-303 (60/60 games)
board size: 19   komi: 7.5
                wins              black         white       avg cpu
lz-minigo-303     46 76.67%       22 73.33%     24 80.00%      8.58
minigo-303        14 23.33%       6  20.00%     8  26.67%     26.36
                                  28 46.67%     32 53.33%

I tested with --parallel_readouts=2 which is slower than the default batch_size of 8 but still faster than --parallel_readouts=1

lz-minigo-303 v minigo-303 (48/60 games)
board size: 19   komi: 7.5
                wins              black         white       avg cpu
lz-minigo-303     41 85.42%       22 91.67%     19 79.17%      9.66
minigo-303         7 14.58%       5  20.83%     2   8.33%     48.23
                                  27 56.25%     21 43.75%

Finally I tested with --parallel_readouts=1 & --c_puct=0.8 (LZ value) which had an even score

lz-minigo-303 v minigo-303 (100/100 games)
board size: 19   komi: 7.5
                wins              black         white       avg cpu
lz-minigo-303     53 53.00%       25 50.00%     28 56.00%      9.91
minigo-303        47 47.00%       22 44.00%     25 50.00%     75.54
                                  47 47.00%     53 53.00%

I'm running two tests now one with just --parallel_readouts=1 and a second with --parallel_readouts=2 --c_puct=0.8

[Ttl]

Shouldn't the equivalent Puct be twice as large in minigo because minigo uses winrate on scale [-1, 1] while LZ uses [0, 1]?

[alreadydone]

If you compile leelaz with USE_TUNER defined
https://github.com/gcp/leela-zero/blob/d362ee86cf7ead85831c8a947c7fed580f63c8eb/src/config.h#L81
you should be able to specify options --puct (0.69 for Minigo) and --fpu_reduction (0 for Minigo). Also LZ uses net_eval for FPU (before reduction). To use get_pure_eval (dynamic parent Q) instead, use my branch patch-4, but revert the changes to fpu_reduction (L275) and puct (L294) in the first commit.

[amj]

@Ttl yes i would expect the equivalent puct to need to be twice as large as well -- very strange that it worked that way.

[sethtroisi]

With @alreadydone's help I enabled USE_TUNER and set --puct 0.69 --fpu_reduction 0
I see very similar move selection now.

disabling rotation I get nearly identical results (I actually have to apply the LZ rotation 1, flip over the X direction)

[fmscole]

How to Converts weights from LZ to minigo ?
Where is the code?

[sethtroisi]

I don't think there's code that converts backwards (LZ to MG) yet.