-
Notifications
You must be signed in to change notification settings - Fork 5
Search and Evaluation parameter Optimization
Optimizing the engine's search and evaluation parameters at the same time by hand is difficult. You can only guess what would be the result if I increase the mobility or king safety and make my late move reduction aggressive and at what max depth do I have to prune the search depth. With the help of auto optimizer this becomes more easier.
This specific optimization uses the bayesian optimization method using UCB or Upper Confidence Bound. Note Lakas has other optimization methods.
--optimizer bayesopt --bo-utility-kind ucb
see command line section below.
The values in the parameter are the engine's default values.
init param: {'FutilityMargin': 60, 'KingAttackWeight': 100, 'LmrFactor': 100, 'MobilityWeight': 100, 'PassedPawnWeightOp': 100, 'QsearchFutilityMargin': 100, 'RazorDepth': 4, 'WinEvalDepth': 7}
- FutilityMargin
- LmrFactor
- QsearchFutilityMargin
- RazorDepth
- WinEvalDepth
- KingAttackWeight
- MobilityWeight
- PassedPawnWeightOp
total budget: 60 (more is better)
games_per_budget: 100 (more is better)
tuning match move control: base_time_sec: 15, inc_time_sec: 0.1, depth=1000
parameter dimension: 8
optimizer: bayesopt, initialization: Hammersley, init_budget: None, middle_point: False, utility_kind: ucb, utility_kappa: 2.576, utility_xi: 0.0, gp_parameters: {'alpha': 0.001, 'normalize_y': True, 'n_restarts_optimizer': 5, 'random_state': None}
python -u lakas.py --optimizer bayesopt --bo-utility-kind ucb --input-data-file bayesopt_ucb_deu_test1.dat --output-data-file bayesopt_ucb_deu_test1.dat --optimizer-log-file log_ucb_deu_test1.txt --base-time-sec 15 --inc-time-sec 0.1 --budget 60 --games-per-budget 100 --concurrency 6 --engine ./engines/deuterium/Deuterium.exe --input-param "{'KingAttackWeight': {'init':100, 'lower':50, 'upper':500}, 'PassedPawnWeightOp': {'init':100, 'lower':50, 'upper':500}, 'WinEvalDepth': {'init':7, 'lower':2, 'upper':9}, 'RazorDepth': {'init':4, 'lower':2, 'upper':6}, 'LmrFactor': {'init':100, 'lower':80, 'upper':120}, 'QsearchFutilityMargin': {'init':100, 'lower':25, 'upper':200}, 'FutilityMargin': {'init':60, 'lower':25, 'upper':100}, 'MobilityWeight': {'init':100, 'lower':50, 'upper':500}}" --opening-file ./start_opening/my_startpos.pgn --opening-file-format pgn
After 60 budgets, at budget 48 it comes up with the following parameter values as best. Optimization time is around 7 hours.
best_param: {'FutilityMargin': 49, 'KingAttackWeight': 120, 'LmrFactor': 92, 'MobilityWeight': 67, 'PassedPawnWeightOp': 98, 'QsearchFutilityMargin': 103, 'RazorDepth': 4, 'WinEvalDepth': 5}
best loss: 0.425 (low is better, it starts at 0.50)
| param | paramtype | default | optimized | |
|---|---|---|---|---|
| 0 | FutilityMargin | search | 60 | 49 |
| 1 | LmrFactor | search | 100 | 92 |
| 2 | QsearchFutilityMargin | search | 100 | 103 |
| 3 | RazorDepth | search | 4 | 4 |
| 4 | WinEvalDepth | search | 7 | 5 |
| 5 | KingAttackWeight | eval | 100 | 120 |
| 6 | MobilityWeight | eval | 100 | 67 |
| 7 | PassedPawnWeightOp | eval | 100 | 98 |
Run a game test between the default and optimized values at TC 30s+100ms. The optimized won by +292 wins after 2000 games.
Note that game simulations in this optimization was done at TC 15s+100ms. See command line section.
Score of Deuterium_optimized vs Deuterium_default: 531 - 339 - 1130 [0.548] 2000
... Deuterium_optimized playing White: 296 - 132 - 572 [0.582] 1000
... Deuterium_optimized playing Black: 235 - 207 - 558 [0.514] 1000
... White vs Black: 503 - 367 - 1130 [0.534] 2000
Elo difference: 33.5 +/- 10.0, LOS: 100.0 %, DrawRatio: 56.5 %
Also a small sample with TC 60s+100ms showed a promising result.
Score of Deuterium_optimized vs Deuterium_default: 27 - 17 - 56 [0.550] 100
... Deuterium_optimized playing White: 11 - 9 - 31 [0.520] 51
... Deuterium_optimized playing Black: 16 - 8 - 25 [0.582] 49
... White vs Black: 19 - 25 - 56 [0.470] 100
Elo difference: 34.9 +/- 45.3, LOS: 93.4 %, DrawRatio: 56.0 %