validation in 0.16 version has significant pause between each match compared to 0.15 #1987
Comments
|
leelaz should start faster if you specify precision in the commands. |
|
@alreadydone Thank you for reply. But why I should specify a precision? I though the leelaz will automatically choose the best precision, am I right? What is more, I just found the -k option is not working in new validation. I use for example
But found no sgf files generated in sgf folder. Is anything wrong with my command line? |
|
I think there is a bug with validation in combination with the binary commands and -k command. I found that you can generate sgf files if you put the -k command to the right of the binary commands, but it doesn't create a folder. |
|
@alreadydone @barrtgt so, should we tag this as a bug? |
|
Precision detection takes time, and you can do it once and then use the detected precision afterwards by specifying it in the commands. It should be possible to not restart leelaz after each game, but the cache and tree would need to be cleared. If specifying precision indeed reduce the pause, I think it's not worth doing. sgf saving is a separate issue. |
|
@alreadydone Thank you so much. I finally understand what you mean. And yes, setting precision explicitly remove the pause. and also putting -k option after two '-- binary' options actually works, the sgf files are generated. However, I found another problem, the genreated sgf file has wrong infomation for White. The White always shows "Human". Something like below
|
|
I think that's because the black leelaz, not autogtp/validation, is responsible for writing the sgf, and has no idea who its opponents is. If you look at the matches on zero.sjeng.org, the sgfs (generated by autogtp instead of validation) don't show white's options, but do show white's network hash, so it may not be so hard to fix this. |
|
To make an engine stronger, calculation speed is only one aspect. In LZ the half precision calculation makes the engine much weaker although more playouts is done per move, as appeared from several games with Phoenixgo. Maybe it's my GPU which doesn't support half precision, but I think with so many calculations for the 40 block weight, the half precision calculation may miss many best moves when the game situation is subtle. That's why I'm wondering whether it is valid to only compare the calculation speed to determine if half precision should be used. |
Citation needed. IIRC when we tested this the influence was negligible. |
AutoGTP fixes up the SGF with the correct opponent, I guess Validation could do it as well. |
|
The last fix shouldn't be difficult but I have to test it. |
|
LZ has no problem to win 2 handicap game with Phoenixgo using single precision. However, LZ lost the following two handicap game with half precision. You would say this is only one game, but LZ is able to win more than 10 single precision games in a row. If we check this half precision game closely, we see the win rate variate wildly, especially at the end of the game. LZ thinks it still has 15% winrate at the very end, while at the middle of the game the winrate even reached 65% (which is justified). If there is such variation I doubt it can find the best moves from time to time. |
|
this behaviour can also happen with leela zero against other bots in ai vs ai, especially when the bots are different, the winrates can dramatically change you may want to gather more statistical data to reach a conclusion |
|
Not trying to draw any conclusion, just want to discuss what is actually going on by looking in detail and compare the decision made by LZ single and LZ half. Note that the quantization error is accumulated and only appear after several moves. Here is again the game I post before at move 231: Click to see the logFrom the exact starting point, LZ-single knows already that the game is now in favor of the opponent and ignores the fake hope H4 which LZ-half sees as 59% win. Instead, LZ-single plays G5 (position A in the picture above) to make sure b F6 is captured. Click to see the log |
|
These are the results for my pc that I tested earlier: Single vs. half precision with net a53 at 10 playouts Have you tried detaching and reattaching multiple times with both single and half on the same position to see if it is consistent? |
|
Yes that was exactly what I have done. I opened two Sabaki windows at the same time though. I would think 10 playouts maybe too short. |
Dear developers,
I tried the new validation program in latest 0.16 release with leelaz 0.16 on ubuntu , and found that it has significant pause between each match compared to previous validation in 0.15 with leelaz 0.15. This is very obvious if we validate two small weight for example No.91 vs No.116. It seems due to that leelaz 0.16 starts much slower than leelaz 0.15, am I right?
best regards
The text was updated successfully, but these errors were encountered: