ELF OpenGo model v2 is released #2212

We train 64-filter, 5-block models from scratch with both AGZ and AZ for 12 hours and compare their final performance. The model trained with AZ wins 100:0 over the model trained with AGZ, indicating that AZ is indeed much more performant.

wonderingabout · 2019-02-13T12:01:59Z

@atoutw
google translate in google chrome too good
i can speak all languages in the world in 1 second 🥇

l1t1 · 2019-02-13T12:56:56Z

https://translate.google.cn/m/translate#auto/

2ji3150 · 2019-02-13T16:18:59Z

Are ELF models able to convert into the quantized ver for LZ? Maybe it can save some traffic. Sinces they are some refenrence points.

Hersmunch · 2019-02-13T19:29:38Z

:)

Silver et al. (2017) establish that during selfplay, the MCTS temperature is set to zero after 30 moves; that is, the policy becomes a Dirac (one-hot) distribution that selects the most visited move from MCTS. However, it is left ambiguous whether a similar temperature decrease is used during training. ELF OpenGo’s training worker uses an MCTS temperature of 1 for all moves (i.e. policy proportional to MCTS visit frequency).

atoutw · 2019-02-13T20:10:36Z

@wonderingabout
Thank you for the translation!!

wonderingabout · 2019-02-13T20:15:31Z

haha, thank you for the thank you 🥇
@atoutw

alreadydone · 2019-02-13T20:46:29Z

A link included in the paper is alive now and contains rich information: http://facebook.ai/developers/tools/elf-opengo
However the blog post is still unviewable now.

Links to all models (network weight files) and training data (selfplay json url)!! https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/urls.csv
Beware not to download more than one file in 3 seconds to avoid getting banned: see rate limit.txt
See selfplay_dataset.txt for details. For 0029000.json.xz which has 2959 games, the selfplay json file is 155 MB. I didn't finish downloading 0000000.json.xz; it should be more than 4 GB. You may want to change https to http to see download progress.

Windows binary of "native" engine: https://dl.fbaipublicfiles.com/elfopengo/play/play_opengo_v2.zip (1.35 GB)

"hawk eye" analysis of historical games, self-play games, analysis with different nets: hawkeye.txt

Ladder test suite: https://github.com/pytorch/ELF/tree/master/ladder_suite

Ishinoshita · 2019-02-13T21:23:09Z

Regarding the first match LZ204 vs Elfv2 on zero.sjeng.org, there is a strange thing, that already happened in the past while matching one of bjiyxjo 15b net against Elfv1: there were two short games (~94 moves) where Elf, and only elf side, played weird moves (second lines or in the middle of nowhere).

In todays match, in 5 games played by the same client 9, Elfv2 plays total crap in the opening, doesn't resign properly (2 games last 400+ moves). Here are two such games:

http://zero.sjeng.org/viewmatch/7b181c46db5a6a4ad1f644b86b80d204bb57037897a8da274da90135c8c418ec?viewer=wgo

Corrupt client ? If not malvolent, why only Elf side each time ? Pb with that client witn the elf converted weight file ? ....

wonderingabout · 2019-02-13T21:24:44Z

@Ishinoshita

i got tired of always mentioning this human behaviour, while no one was willing to take measures for it, so ... same as usual ?

too bad, but what can i do except moving on ?

Ishinoshita · 2019-02-13T21:32:51Z

Same thing for 10 games played by client 23 in this match elfv1 vs elfv2. But this time, both sides play crap moves, what matches my hypothesis that there is a clien out there that screws only with elf% weights.

l1t1 · 2019-02-14T00:01:54Z

@wonderingabout did you test the native engine vs lz ?

l1t1 · 2019-02-14T02:37:41Z

paper updated https://arxiv.org/abs/1902.04522v2

sethtroisi · 2019-02-14T03:42:12Z

Very minor edits. Some spacing changes on first page

page 12:
Asynchronized (AlphaZero) mode

The asynchronous approach achieves
over 5x the selfplay throughput of the synchronous approach
on our hardware setup. (2) The ratio of selfplay games to
training minibatches increased by roughly 1.5x [inserted], thus helping
to prevent overfitting.

alreadydone · 2019-02-14T03:53:10Z

I reported the missing value on their issues. By the way, the blog post is now accessible: https://ai.facebook.com/blog/open-sourcing-new-elf-opengo-bot-and-go-research/
just read this:

we're gratified to see players in the Go community use it to hone their skills and study the game.
"I can definitely say that the ELF OpenGo project has brought a huge impact on the Korean Go community,” said Beomgeun Cho, Assistant Director of PR, Korea Baduk Association. “Since it came out, almost every competitive professional player in Korea has been using the ELF Go program to analyze their own and other players’ games. And because of that, not only has the level of Korean Go improved, but the level of the whole world has been improved significantly.”

:)

wonderingabout · 2019-02-14T07:02:53Z

@l1t1

no, not yet, i definitely would like to, and more importantly see if native elf can finally be exported to gtp2ogs

l1t1 · 2019-02-14T07:08:02Z

i try to use gogui, but it didnt work under

D:\tool\go_gui>D:\tool\go_gui\gogui-twogtp -black "D:\elf_cpu_full\elf\df_console.exe " -white "D:\leela-zero-0.16-win64\leelaz.exe --gtp -w D:\elfv2.gz --noponder -v 3201"  -games 10 -sgffile fb_elfv2 -auto -komi 7.5

wonderingabout · 2019-02-14T07:10:10Z

i think you need to read the documentation, but no idea what it says @l1t1

l1t1 · 2019-02-14T07:16:38Z

elf's play_opengo_v2.zip has a readme.pdf it says

Step 1: Double click the “run.bat” to start Sabaki (we use Sabaki version 35 as our front-end);
Step 2: Start a new game: File (left-top) → New → select either “black” or “white” as the ELF AI to play.
The GPU version takes on average 6 seconds per move on a high-end NVIDIA GPU, and 40 seconds per
move on a high-end Intel CPU.
Optional
In case you have GTP connection problems, you may check if the path of the AI engine is set to the
df_console.exe in our ELF local folder.

wonderingabout · 2019-02-14T07:42:10Z

i'm too busy to look at this atm sorry @l1t1

i hope other people who tried the windows native pre built binary can help you with their feedback

hanamichigo · 2019-02-14T08:17:27Z

df_console.exe only works at sabaki 35. probably because of their limited list_commands (their gtp version is 2). i currently use sabaki 35 to play it against various minigo and lz versions.

you may also tinker with the available parameters in df_console.exe --help. one thing i notice is that it doesn't support multi-gpu. so far i'm unable to make it print the pv and winrates per move, maybe someone can figure it out.

l1t1 · 2019-02-14T08:28:43Z

from elf readme. of https://github.com/pytorch/ELF
have someone make it works?

Running a Go bot

Here is a basic set of commands to run and play the bot via the GTP protocol:

Build ELF and run source scripts/devmode_set_pythonpath.sh as described above.
Train a model, or grab a pretrained model.
Change directory to scripts/elfgames/go/
Run ./gtp.sh path/to/modelfile.bin --verbose --gpu 0 --num_block 20 --dim 256 --mcts_puct 1.50 --batchsize 16 --mcts_rollout_per_batch 16 --mcts_threads 2 --mcts_rollout_per_thread 8192 --resign_thres 0.05 --mcts_virtual_loss 1

teraStars · 2019-02-15T02:44:20Z

I have tested 2 mathes, LZ 05d10f27 vs ELFv2.
ELFv2 both failed.

l1t1 · 2019-02-15T03:04:51Z

elfv2 p800 win lz204 on cgos
http://www.yss-aya.com/cgos/19x19/cross/LZ_05db_ELFv2_p800.html

alreadydone · 2019-02-15T06:29:51Z

ELFv2 seems very keen on playing out Mi Yuting's Dagger (小芈飞刀, [a recent example of which is in LG championship game 2) but does not always correctly handle it, from what I heard.

alreadydone · 2019-02-16T02:18:01Z

ELFv2 (model 1290) self-play white winrate: 58.6% (7810/13319) (the model chosen for release based on pairwise model comparisons)
ELF model 1499 self-play white winrate: 67.1% (9855/14690) (the second to last model; the last model didn't play any games)
From:
https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/selfplays/json_xz/1290000.json.xz
https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/selfplays/json_xz/1499000.json.xz

l1t1 · 2019-02-16T02:28:46Z

@alreadydone the text elfv2 and elf is a bit mislead
do you verify https://github.com/pytorch/ELF/releases/download/pretrained-go-19x19-v2/pretrained-go-19x19-v2.bin and https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/models/1228000.bin are same.
as my test, v2.bin 's structure is not same as 1500000.bin (pytorch/ELF#133 (comment))

l1t1 · 2019-02-16T02:29:23Z

typo 1228 ->1290

alreadydone · 2019-02-16T02:39:19Z

I just verified CRC32 = 3E459311 for both pretrained-go-19x19-v2.bin and https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/models/1290000.bin.
It's in the paper:

Based on pairwise model comparisons, we extract our “final” model from the main training run after 1.29 million training minibatches.

l1t1 · 2019-02-16T02:44:58Z

if you have windows pc, fc /b is ok, nevertheless, crc is also good

l1t1 · 2019-02-16T02:49:05Z

@alreadydone you quote is right, but the next statement isn

The ﬁnal model is approximately 150 ELO
stronger than the prototype model,

alreadydone · 2019-02-16T02:56:43Z

prototype is ELFv0. 150 Elo is maybe not too far off?

thanks I didn't know about fc /b. I just compress with WinRAR and check CRC32.

l1t1 · 2019-02-16T03:02:28Z

i read the paper again, and I see, prototype means elfv0
elo from https://zero.sjeng.org//data/elograph.json, but elfv0 isnt so close to elfv1 as i remember

elfv0 {"rating":12383,"net":9779286.12383,"sprt":"TEST","hash":"62b541"
elfv1 {"rating":12387,"net":9779286.12387,"sprt":"TEST","hash":"d13c40",
elfv2 {"rating":13572,"net":12468986.13572,"sprt":"TEST","hash":"05dbca",

alreadydone · 2019-02-16T03:14:55Z

A net's rating on zero.sjeng.org is probably anchored to its last opponent (I think the net needs to appear to the left and its opponent to the right for the match to adjust the net's rating), so it only takes a single match into account, and the result can depend on luck and be unreliable ... There's no Bradley-Terry model implemented as far as I know.

roy7 · 2019-02-16T03:28:22Z

Yes @alreadydone is exactly correct. Minigo has the fancy eval system in place, we just have a very basic A-B comparison where we always compare against the current best network and chain the promotions together to have a sense of progress.

l1t1 · 2019-02-16T03:54:34Z

https://cloudygo.com/ringmaster/ has some leelazero crosstable

sethtroisi · 2019-02-16T10:58:38Z

https://cloudygo.com/leela-zero-v3/eval-graphs has a better setup for crosstable.
Each model is played against a panel of models so you see less of the chain of promotions that leads to self-play elo inflation.

alreadydone · 2019-02-17T01:23:35Z

@sethtroisi I don't see any ELF models in your link. Should we convert some ELF models and match them against Minigo v15 because 20b/19b models are more comparable (at equal playouts)? The best models can be seen from the paper (see plot below) and downloaded from https://dl.fbaipublicfiles.com/elfopengo/v2_training_run/urls.csv.

l1t1 · 2019-02-20T21:50:09Z

https://dl.fbaipublicfiles.com/elfopengo/play/play_opengo_v2.zip updated

https://www.lifein19x19.com/viewtopic.php?f=18&t=16441&sid=f5d415fb55b0f624d608e05c8815c1f7&start=40

l1t1 · 2019-02-20T21:51:01Z

https://www.lifein19x19.com/viewtopic.php?p=241852#p241852

l1t1 mentioned this issue Feb 13, 2019

could you test elfv2 vs minigo v15 m990 sethtroisi/cloudygo#44

Closed

Navigation Menu

ELF OpenGo model v2 is released #2212

ELF OpenGo model v2 is released #2212

Comments

hyln9 commented Feb 13, 2019

l1t1 commented Feb 13, 2019

herazul commented Feb 13, 2019

hyln9 commented Feb 13, 2019

alreadydone commented Feb 13, 2019

l1t1 commented Feb 13, 2019

l1t1 commented Feb 13, 2019

sethtroisi commented Feb 13, 2019

l1t1 commented Feb 13, 2019

l1t1 commented Feb 13, 2019

wonderingabout commented Feb 13, 2019

wonderingabout commented Feb 13, 2019 • edited

sethtroisi commented Feb 13, 2019

wonderingabout commented Feb 13, 2019 • edited

atoutw commented Feb 13, 2019

wonderingabout commented Feb 13, 2019 • edited

barrtgt commented Feb 13, 2019

wonderingabout commented Feb 13, 2019 • edited

l1t1 commented Feb 13, 2019

2ji3150 commented Feb 13, 2019 • edited

Hersmunch commented Feb 13, 2019

atoutw commented Feb 13, 2019

wonderingabout commented Feb 13, 2019

alreadydone commented Feb 13, 2019 • edited

Ishinoshita commented Feb 13, 2019

wonderingabout commented Feb 13, 2019 • edited

Ishinoshita commented Feb 13, 2019

l1t1 commented Feb 14, 2019

l1t1 commented Feb 14, 2019

sethtroisi commented Feb 14, 2019

alreadydone commented Feb 14, 2019

wonderingabout commented Feb 14, 2019

l1t1 commented Feb 14, 2019

wonderingabout commented Feb 14, 2019

l1t1 commented Feb 14, 2019

wonderingabout commented Feb 14, 2019

hanamichigo commented Feb 14, 2019

l1t1 commented Feb 14, 2019

teraStars commented Feb 15, 2019

l1t1 commented Feb 15, 2019

alreadydone commented Feb 15, 2019

alreadydone commented Feb 16, 2019 • edited

l1t1 commented Feb 16, 2019

l1t1 commented Feb 16, 2019

alreadydone commented Feb 16, 2019 • edited

l1t1 commented Feb 16, 2019

l1t1 commented Feb 16, 2019

alreadydone commented Feb 16, 2019 • edited

l1t1 commented Feb 16, 2019

alreadydone commented Feb 16, 2019

roy7 commented Feb 16, 2019

l1t1 commented Feb 16, 2019

sethtroisi commented Feb 16, 2019 • edited

alreadydone commented Feb 17, 2019

l1t1 commented Feb 20, 2019

l1t1 commented Feb 20, 2019

wonderingabout commented Feb 13, 2019 •

edited

wonderingabout commented Feb 13, 2019 •

edited

wonderingabout commented Feb 13, 2019 •

edited

wonderingabout commented Feb 13, 2019 •

edited

2ji3150 commented Feb 13, 2019 •

edited

alreadydone commented Feb 13, 2019 •

edited

wonderingabout commented Feb 13, 2019 •

edited

alreadydone commented Feb 16, 2019 •

edited

alreadydone commented Feb 16, 2019 •

edited

alreadydone commented Feb 16, 2019 •

edited

sethtroisi commented Feb 16, 2019 •

edited