-
Notifications
You must be signed in to change notification settings - Fork 529
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Update q to cp conversion formula #1193
Conversation
Revert the formula to PR841 equation.
It seems that after every TCEC season we start to retune our formula as it doesn't fit the centipawn scale well. I guess what's really happen is different networks have different Q scale so to say, meaning that different networks will need different tunes. Maybe it makes sense to have a library of a few hundreds positions with SF eval and run Lc0 evals periodically on them to re-tune the formula.. |
Don't merge this just yet, I'll tune this further when there are slightly more games played in sufi, |
I think this formula is much better than our current one for 2 reasons: |
Based on the current TCEC games, the Q to centipawn conversion formula seems to be too pessimistic at high Q. lc0 cp evaluations are often much lower than the other engines.
I downloaded the TCEC SUFI evaluations and compared to the lc0 cp evaluations to SF evaluations. Below are the results:
The evaluation for Q < 0 comes from only one game and can be ignored. The current formula is the line labeled with centipawn. The first too optimistic conversion function is centipawn_2018 line. I also plotted the conversion function from #841 that used the old equation but fitted to more games. The #841 equation seems to fit very well to the median SF evaluation. I can get slightly better fit with more complex functions, but this one has advantage of being easily invertible which is nice for calculating the Q from centipawns. This PR reverts the conversion function to one from #841.
The average error to median SF eval on this dataset is 0.79 pawns with the current function, 0.28 with this function and 1.85 with the centipawn_2018.
The script for downloading evaluations and plotting the graphs can be found at: https://gist.github.com/Ttl/9926f70800b3fbd7314c150108d4ba61