Skip to content

Optimization Comparison

Jump to bottom
fsmosca edited this page Dec 12, 2020 · 19 revisions

Conditions

The loss is just the result of the optimizer param against the default param in a match of 1k games. If optimizer param won by a score of 0.51 from (wins+draw/2)/games, the loss that will be sent to the optimizer will be 1.0 - 0.51 or 0.49 as the optimizer tries to minimize it. Stockfish engine is used in this optimization.

Optimization

  • Budgets: 100
  • Games/budget: 1000
  • Game move control: depth 6

Game Verification Test

  • Games: 10k
  • Game move control: depth 6
  • Optimized param vs default param

Parameters to be optimized

--input-param "{'RazorMargin': {'init':527, 'lower':100, 'upper':700}, 'KingAttackWeights[2]': {'init':81, 'lower':0, 'upper':150}, 'eMobilityBonus[0][7]': {'init':20, 'lower':-20, 'upper':50}, 'mMobilityBonus[1][7]': {'init':63, 'lower':-50, 'upper':150}, 'eMobilityBonus[3][21]': {'init':168, 'lower':50, 'upper':250}, 'eThreatByRook[1]': {'init':46, 'lower':0, 'upper':150}, 'mThreatByMinor[3]': {'init':77, 'lower':0, 'upper':150}, 'eThreatByKing': {'init':89, 'lower':10, 'upper':150}, 'eThreatByPawnPush': {'init':39, 'lower':0, 'upper':100}}"

Default param values

init param: {'KingAttackWeights[2]': 81, 'RazorMargin': 527, 'eMobilityBonus[0][7]': 20, 'eMobilityBonus[3][21]': 168, 'eThreatByKing': 89, 'eThreatByPawnPush': 39, 'eThreatByRook[1]': 46, 'mMobilityBonus[1][7]': 63, 'mThreatByMinor[3]': 77}

Sample command line

python -u lakas.py --optimizer oneplusone --output-data-file oneplusone_t1.dat --optimizer-log-file opt_log_t1.txt --depth 6 --base-time-sec 30 --budget 100 --games-per-budget 1000 --concurrency 6 --engine ./engines/stockfish-modern/stockfish.exe --input-param "{'RazorMargin': {'init':527, 'lower':100, 'upper':700}, 'KingAttackWeights[2]': {'init':81, 'lower':0, 'upper':150}, 'eMobilityBonus[0][7]': {'init':20, 'lower':-20, 'upper':50}, 'mMobilityBonus[1][7]': {'init':63, 'lower':-50, 'upper':150}, 'eMobilityBonus[3][21]': {'init':168, 'lower':50, 'upper':250}, 'eThreatByRook[1]': {'init':46, 'lower':0, 'upper':150}, 'mThreatByMinor[3]': {'init':77, 'lower':0, 'upper':150}, 'eThreatByKing': {'init':89, 'lower':10, 'upper':150}, 'eThreatByPawnPush': {'init':39, 'lower':0, 'upper':100}}" --opening-file ./start_opening/ogpt_chess_startpos.epd

Optimizers

1. OneplusOne

--optimizer oneplusone

best: option.KingAttackWeights[2]=36 option.RazorMargin=568 option.eMobilityBonus[0][7]=6 option.eMobilityBonus[3][21]=192 option.eThreatByKing=110 option.eThreatByPawnPush=56 option.eThreatByRook[1]=111 option.mMobilityBonus[1][7]=69 option.mThreatByMinor[3]=2

2. TBPSA

--optimizer tbpsa

best: option.KingAttackWeights[2]=71 option.RazorMargin=504 option.eMobilityBonus[0][7]=12 option.eMobilityBonus[3][21]=194 option.eThreatByKing=101 option.eThreatByPawnPush=39 option.eThreatByRook[1]=41 option.mMobilityBonus[1][7]=55 option.mThreatByMinor[3]=79

3. BayesOpt ei default

BayesOpt = Bayessian Optimization --optimizer bayesopt --bo-utility-kind ei

best: option.KingAttackWeights[2]=0 option.RazorMargin=636 option.eMobilityBonus[0][7]=-20 option.eMobilityBonus[3][21]=148 option.eThreatByKing=84 option.eThreatByPawnPush=52 option.eThreatByRook[1]=35 option.mMobilityBonus[1][7]=126 option.mThreatByMinor[3]=83

4. BayesOpt poi default

--optimizer bayesopt --bo-utility-kind poi

best: option.KingAttackWeights[2]=87 option.RazorMargin=509 option.eMobilityBonus[0][7]=23 option.eMobilityBonus[3][21]=137 option.eThreatByKing=61 option.eThreatByPawnPush=53 option.eThreatByRook[1]=22 option.mMobilityBonus[1][7]=58 option.mThreatByMinor[3]=55

5. BayesOpt ucb

--optimizer bayesopt --bo-utility-kind ucb

best: option.KingAttackWeights[2]=77 option.RazorMargin=283 option.eMobilityBonus[0][7]=-9 option.eMobilityBonus[3][21]=105 option.eThreatByKing=114 option.eThreatByPawnPush=41 option.eThreatByRook[1]=47 option.mMobilityBonus[1][7]=56 option.mThreatByMinor[3]=47

6. SPSA

--optimizer spsa

best: option.KingAttackWeights[2]=82 option.RazorMargin=498 option.eMobilityBonus[0][7]=15 option.eMobilityBonus[3][21]=150 option.eThreatByKing=85 option.eThreatByPawnPush=30 option.eThreatByRook[1]=54 option.mMobilityBonus[1][7]=55 option.mThreatByMinor[3]=80

7. BayesOpt ei exploit

Exploit: --bo-utility-xi 0.0001
--optimizer bayesopt --bo-utility-kind ei --bo-utility-xi 0.0001

best: option.KingAttackWeights[2]=32 option.RazorMargin=427 option.eMobilityBonus[0][7]=13 option.eMobilityBonus[3][21]=157 option.eThreatByKing=108 option.eThreatByPawnPush=47 option.eThreatByRook[1]=37 option.mMobilityBonus[1][7]=66 option.mThreatByMinor[3]=4

8. BayesOpt ei explore

Explore: --bo-utility-xi 10000
--optimizer bayesopt --bo-utility-kind ei --bo-utility-xi 10000

best: option.KingAttackWeights[2]=78 option.RazorMargin=537 option.eMobilityBonus[0][7]=-3 option.eMobilityBonus[3][21]=156 option.eThreatByKing=94 option.eThreatByPawnPush=29 option.eThreatByRook[1]=18 option.mMobilityBonus[1][7]=55 option.mThreatByMinor[3]=93

9. BayesOpt poi exploit

Exploit: --bo-utility-xi 0.0001
--optimizer bayesopt --bo-utility-kind poi --bo-utility-xi 0.0001

best: option.KingAttackWeights[2]=51 option.RazorMargin=100 option.eMobilityBonus[0][7]=16 option.eMobilityBonus[3][21]=250 option.eThreatByKing=116 option.eThreatByPawnPush=36 option.eThreatByRook[1]=68 option.mMobilityBonus[1][7]=45 option.mThreatByMinor[3]=146

10. BayesOpt poi explore

Explore: --bo-utility-xi 10000
--optimizer bayesopt --bo-utility-kind poi --bo-utility-xi 10000

best: option.KingAttackWeights[2]=57 option.RazorMargin=100 option.eMobilityBonus[0][7]=5 option.eMobilityBonus[3][21]=235 option.eThreatByKing=69 option.eThreatByPawnPush=38 option.eThreatByRook[1]=53 option.mMobilityBonus[1][7]=63 option.mThreatByMinor[3]=57

11. BayesOpt ucb exploit

--optimizer bayesopt --bo-utility-kind ucb --bo-utility-kappa 0.0001

best: option.KingAttackWeights[2]=1 option.RazorMargin=428 option.eMobilityBonus[0][7]=8 option.eMobilityBonus[3][21]=201 option.eThreatByKing=103 option.eThreatByPawnPush=36 option.eThreatByRook[1]=30 option.mMobilityBonus[1][7]=-18 option.mThreatByMinor[3]=50

12. BayesOpt ucb explore

--optimizer bayesopt --bo-utility-kind ucb --bo-utility-kappa 10000

best: option.KingAttackWeights[2]=87 option.RazorMargin=440 option.eMobilityBonus[0][7]=14 option.eMobilityBonus[3][21]=155 option.eThreatByKing=67 option.eThreatByPawnPush=67 option.eThreatByRook[1]=25 option.mMobilityBonus[1][7]=-33 option.mThreatByMinor[3]=66

13. NGOpt

--optimizer ngopt

best: option.KingAttackWeights[2]=44 option.RazorMargin=527 option.eMobilityBonus[0][7]=16 option.eMobilityBonus[3][21]=186 option.eThreatByKing=123 option.eThreatByPawnPush=32 option.eThreatByRook[1]=57 option.mMobilityBonus[1][7]=63 option.mThreatByMinor[3]=77

14. CMAES

--optimizer cmaes

best: option.KingAttackWeights[2]=116 option.RazorMargin=489 option.eMobilityBonus[0][7]=20 option.eMobilityBonus[3][21]=220 option.eThreatByKing=119 option.eThreatByPawnPush=46 option.eThreatByRook[1]=60 option.mMobilityBonus[1][7]=86 option.mThreatByMinor[3]=83

Result table

Each test has 10k games at depth 6, between default and best param from the specified optimizer.

TestNum Optimizer UtilityKind xi kappa Elo Error
1 OnePlusOne - - - -8.1 +/- 6.3
2 TBPSA - - - +0.6 +/- 6.3
3 BayesOpt default ei 0.0 - -5.9 +/- 6.3
4 BayesOpt default poi 0.0 - +1.8 +/- 6.3
5 BayesOpt default ucb - 2.576 +3.9 +/- 6.3
6 SPSA - - - +0.5 +/- 6.3
7 BayesOpt exploit ei 0.0001 - +4.3 +/- 6.3
8 BayesOpt explore ei 10000 - -2.1 +/- 6.3
9 BayesOpt exploit poi 0.0001 - -3.6 +/- 6.3
10 BayesOpt explore poi 10000 - +5.1 +/- 6.3
11 BayesOpt exploit ucb - 0.001 -0.7 +/- 6.3
12 BayesOpt explore ucb - 10000 -2.3 +/- 6.3
13 NGOpt - - - -2.6 +/- 6.3
14 CMAES - - - -0.3 +/- 6.3

References

  • BayesianOptimization
    • poi: An acquisition method that stands for Point of Improvement.
    • ei: An acquisition method that stands for Expected Improvement.
    • ucb: An acquisition method that stands for Upper Confidence Bounds.
Clone this wiki locally