-
Notifications
You must be signed in to change notification settings - Fork 526
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Improvement of backendbench assistant. #1458
Conversation
src/benchmark/backendbench.cc
Outdated
|
||
if (nps > best_nps) { | ||
if (best==0 || | ||
((i*1.0f/best*1.0f-1.0f)*threshold < nps/best_nps-1.0f)) { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You can rewrite this as (i-best)*threshold*best_nps < (nps-best_nps)*best
.
Then you can initialize best to 1 to avoid the best==0 check and the nested if
.
src/benchmark/backendbench.cc
Outdated
@@ -79,8 +75,7 @@ void BackendBenchmark::Run() { | |||
options.Add<IntOption>(kMaxBatchSizeId, 1, 1024) = 256; | |||
options.Add<StringOption>(kFenId) = ChessBoard::kStartposFen; | |||
options.Add<BoolOption>(kClippyId) = false; | |||
options.Add<FloatOption>(kClippyThresholdId, 0.0f, 1.0f) = 0.05f; | |||
options.Add<FloatOption>(kClippyToleranceId, 0.0f, 1.0f) = 0.03f; | |||
options.Add<FloatOption>(kClippyThresholdId, 0.0f, 1.0f) = 0.1f; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You suggestion to change the default to 0.2 seems reasonable,
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
0.2 would be my best bet for max. playing strength, favoring lower batch sizes. Can be changed later, if someone conducts more tests.
Current logic is to replace batchsize with highest nps, if new nps is 1+threshold times higher than previous best.
However, it is known that generally higher batchsize numbers perform worse in gameplay, e.g. a 4x higher batchsize with 1.05x higher nps shouldn't be accepted while a 1.10x higher batchsize with 1.04x higher nps is most probably better.
Now the acceptance rule is based on a linear function requiring +10% nps (compared to previous best) for each +100% of batchsize (to previous best).
Example:
Edit: Based on some short test matches I suspect that the default threshold can be increased from 0.1 to 0.2 (+15% nps with 2x batchsize was weaker in a test):