What does it mean that won 200 games against LZ on "default settings"?

[bochen2027]

ELF OpenGo has been successful playing against both other open source bots and human Go players. We played and won 200 games against LeelaZero (158603eb, Apr. 25, 2018), the strongest publicly available bot, using its default settings and no pondering.

Does this mean that only 3200 visits were used? By default settings I'm assuming this is referring to LZ's own match games settings? Can we get more details on the exact specifications and hardware used for both sides for these matches? And when stated that it played and won 200 matches, is that means it won all 200 matches that it played, or that it simply won a total number of 200 matches against an unknown number of total matches played? Can you publish the sgf for these 200 games?

Can you advise as to how strong is the raw network on 1 single playout? (can you release a binary executable for Windows, Linux?)

edit: Has the raw training data of the self played games been published?

The release article mentioned one of the objectives of this openness is to help community projects such as Leela Zero, I see no better way of doing that then releasing the raw played games and allowing LZ to immediately train on them to get stronger. (assuming it was a 200:0 match and on objective parity conditions, and not like the Deepmind AlphaZero vs Stockfish chess shenanangians)

[kityanhem]

Same question but asking for Kifu 😄

[jma127]

Hi @Hydrogenpi, thanks for your interest! We normalized both ELF OpenGo and LeelaZero to use the same thinking time per move (50 seconds) on a V100 GPU.

LeelaZero's visit count seemed to vary from move to move. Based on a random sample of 8 games, it had an average of around 43000 rollouts per move, going as low as 14000 rollouts per move and as high as 231000 rollouts per move.

ELF OpenGo had a fixed budget of 80000 rollouts per move.

[bochen2027]

@jma127 Thanks for the open sourcing. It can only serve to help the community of Go and maybe force Google's hand to release the AGZ weights so we can use them and compare. In any case, if your team has done any scalability tests for example multiple V100's up to 4x, 8x, or even more and you have data describing how well it scales if that info can be somehow shared that would be great as well. Its common knowledge anyone can rent V100 on aws for a few bucks/hour these days, essentially giving superhuman access of Go to anyone in the world. So it would be useful to know if fb team has already tested it on multiple V100 to see how much increment strength it gains.

Looking forward to the promised refinements in the weeks/months ahead! Great good work and thanks for sharing and helping out the worldwide Go effort... I have always believed openness and true democracy of AI is very important, and although in this narrow case Google/Deepmind got there first with AG/AGZ, general availability is also just as important, the fact that facebook stepped up to the plate to help the Go community whereas Google either would not and/or could not, it speaks volumes!

My initial reaction to today's wonderful news:

"One benefit of pushing forward open bots that are superhuman is it may force the hand of others as well, now that Fb has open sourced their bot that they claim is stronger than current LZ, maybe Deepmind will come back for thirds to get one last PR hooray for Google by open sourcing their AGZ weights to spit in fb's face. open competition is good for go. This is truly the end of a human era."

[diadorak]

Do you use a 20 blocks times 224 filters network?

[jma127]

@Hydrogenpi, thank you for the kind words. Addressing your edited questions:

• We had ELF OpenGo and LeelaZero play exactly 200 matches, with a 200:0 outcome.
• Most of our team is away on travel but we'll strive to publish these 200 game records soon.
• We're a small team, so one of our hopes in open-sourcing the project is that interested community members will create various interfaces and executables with ELF OpenGo. Right now though, it should be possible to use many GTP-compatible Go interfaces to play with ELF OpenGo on a compatible system.
• Publishing all the selfplay games requires a bit more infrastructure than we have right now, but we'll certainly put this on our roadmap.

[jma127]

@diadorak Yes, the network is a 224 filter x 20 block ResNet.

[WYVYW]

How much self-play games does ELF have (or did it have before reaching its flatline)?

[jma127]

Closing this issue for now -- we intend to release a much more detailed technical report of our work soon, so stay tuned. Thanks again for the interest, everyone :)