-
Notifications
You must be signed in to change notification settings - Fork 9
Distilled Networks
I have completed the 32930 net distilled to 112x9-se and the Ender128-90l net distilled to 112x9-se.
Enjoy.
Once I have the sweet spot, I'll need to optimize the distillation process and then distill Ender as well. I also have some search changes that will help on low nodes and cpu. My goal is to beat Ethereal 1CPU. Still about 200-250 elo short.
In an attempt to find the dimensions that work best on CPU, I've been distilling the test 10 network, ID11258 down to various sized se networks. (The version of the training code I’m using for distillation can be found here.)
- 11258-16x2-se
- 11258-24x3-se
- 11258-32x4-se
- 11258-48x5-se
- 11258-64x6-se
- 11258-80x7-se
- 11258-96x8-se
- 11258-104x9-se
- 11258-112x9-se
- 32930-112x9-se
- ender-112x9-se
- 11258-120x9-se
- 11258-128x9-se
- 11258-112x10-se
- 11258-120x10-se
- 11258-128x10-se - both the regular and swa nets
Other distilled networks:
And scs-ben’s other small networks.
Also, the old "main line" nets are a bit hard to find. I have a pb.gz version of 598 here.
I distilled a 200x20-se network under the assumption that it would be as strong as the 256x20 and faster. Well, not quite. Maybe it needs to be a little bigger. 216x20? Maybe someone can figure out what to do with it.
# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L D(%)
1 ID11258 : 0 22 22.0 40 55.0 94 5 34 1 85.0
2 ID11258-200x20-se-10 : -35 22 18.0 40 45.0 --- 1 34 5 85.0
To support my hunt for the best dimensions, I've narrowed the tournament to just 11258 derived nets and anchor engines.
# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L D(%)
1 ethereal : 3256 36 419.5 477 87.9 100 375 89 13 18.7
2 ID11258-112x9-se : 2988 26 301.0 497 60.6 64 203 196 98 39.4
3 ID11258-120x9-se : 2981 29 222.0 369 60.2 57 153 138 78 37.4
4 ID11258-104x9-se : 2977 30 230.5 388 59.4 51 161 139 88 35.8
5 ID11258-80x7-se : 2977 30 206.5 348 59.3 69 139 135 74 38.8
6 ID11258-112x10-se : 2966 29 229.5 392 58.5 51 156 147 89 37.5
7 ID11258-64x6-se : 2966 27 232.0 399 58.1 58 162 140 97 35.1
8 ID36092 : 2961 33 183.5 319 57.5 55 121 125 73 39.2
9 ID11258-128x9-se : 2958 30 187.5 331 56.6 71 123 129 79 39.0
10 crafty25.2 : 2946 26 268.5 474 56.6 51 216 105 153 22.2
11 ID11258-120x10-se : 2946 26 269.5 476 56.6 52 188 163 125 34.2
12 ID11258-96x8-se-5 : 2945 26 271.0 479 56.6 55 188 166 125 34.7
13 cheng4 : 2942 28 200.5 376 53.3 60 147 107 122 28.5
14 ID11258-128x10-se : 2937 26 250.5 453 55.3 92 167 167 119 36.9
15 ID11258-48x5-se : 2909 29 194.0 383 50.7 75 130 128 125 33.4
16 amoeba : 2895 29 183.5 387 47.4 94 132 103 152 26.6
17 ID50417 : 2850 49 61.0 137 44.5 98 34 54 49 39.4
18 ID11258-32x4-se : 2787 30 132.0 374 35.3 99 80 104 190 27.8
19 winter : 2731 31 109.5 395 27.7 89 70 79 246 20.0
20 ID11258-24x3-se : 2703 31 111.5 423 26.4 100 75 73 275 17.3
21 crafty19.18 : 2591 35 85.5 482 17.7 100 52 67 363 13.9
22 ID11258-16x2-se-4 : 2408 52 28.0 395 7.1 --- 13 30 352 7.6
White advantage = 45.74 +/- 4.89
Draw rate (equal opponents) = 38.06 % +/- 0.90
For a sense of strength, here a tournament, 2+2, Noomen 3, played twice, no tb. (Used ccrl loose anchors for ab engines.)
# PLAYER : RATING ERROR POINTS PLAYED (%) CFS(%) W D L D(%)
1 ethereal : 3260 40 451.0 502 89.8 100 412 78 12 15.5
2 ID11258-112x9-se : 2979 35 253.5 407 62.3 90 180 147 80 36.1
3 crafty25.2 : 2952 31 339.0 540 62.8 64 273 132 135 24.4
4 ID11248-128x10-se : 2946 29 338.5 566 59.8 52 251 175 140 30.9
5 ID36092 : 2945 36 199.0 332 59.9 63 131 136 65 41.0
6 ID35689 : 2937 35 217.5 360 60.4 57 152 131 77 36.4
7 ID11258-96x8-se-5 : 2934 31 281.0 490 57.3 72 197 168 125 34.3
8 ID35975 : 2923 32 251.0 436 57.6 57 171 160 105 36.7
9 ID36089 : 2919 36 182.0 321 56.7 54 111 142 68 44.2
10 cheng4 : 2917 37 164.0 293 56.0 67 124 80 89 27.3
11 ID11258-128x10-se : 2905 44 114.0 203 56.2 52 78 72 53 35.5
12 ID11258-128x10-se-swa : 2904 56 44.0 79 55.7 64 29 30 20 38.0
13 amoeba : 2891 34 170.5 332 51.4 68 116 109 107 32.8
14 scs-64x8-run1-550000 : 2881 36 155.0 307 50.5 83 106 98 103 31.9
15 192x10-se-net58 : 2857 39 133.0 255 52.2 100 90 86 79 33.7
16 ID11248-256x12-se : 2743 39 106.5 307 34.7 60 54 105 148 34.2
17 winter : 2737 35 104.5 318 32.9 53 64 81 173 25.5
18 ID9149 : 2735 41 100.0 267 37.5 57 62 76 129 28.5
19 ID11258-24x3-se : 2712 250 0.5 2 25.0 51 0 1 1 50.0
20 ID11258-24x3-se-swa : 2706 259 0.5 2 25.0 74 0 1 1 50.0
21 ID11258 : 2615 43 68.5 284 24.1 65 39 59 186 20.8
22 crafty19.18 : 2605 36 123.5 539 22.9 100 79 89 371 16.5
23 ID11258-16x2-se-3 : 2453 53 38.0 287 13.2 52 19 38 230 13.2
24 ID11258-16x2-se-4 : 2451 50 40.0 321 12.5 --- 18 44 259 13.7
White advantage = 42.08 +/- 5.20
Draw rate (equal opponents) = 39.19 % +/- 0.97
First, let me point you at a decent article.
A brief discription:
When you train a network, you feed it the history and position info and the value (game outcome) and policy from the move made (supervised learning) or the policy values from the 800 node search. In distilling, you use regular training data for history and position info, but value and policy comes from the teacher network. And yes, you could distill to a bigger network.
You will need a recent lc0 executable to run the nets. You can find v0.20.1 here
If you are running on cpu, you want the blas flavor.
As far as setting it up, it’s a uci engine. Point your GUI at it. The three uci options worth changing are:
option name WeightsFile type string default <autodiscover>
You’ll want to point it at wherever you placed the .pb.gz weights file.
option name DirichletNoise type check default false
You’ll want to set this to true to get a little variety in play.
option name SyzygyPath type string default
You want a semicolon separated list of dirs where you keep your endgame tablebases.
Best of luck.
My new (old) blog is at lczero.libertymedia.io