A way to make 19x19 weights fit for other board sizes

[hzyhhzy]

Here are some weights from 9x9 to 25x25.(converted from #205(85a93684) lz weight),

https://drive.google.com/open?id=1jQjCbK1xWGBglD402WpBMyeEieDG-h83

and a very rough CPP code.

https://drive.google.com/open?id=1bgkVBeCIwTgA88DG07sQE8oXHUdDuqt7

I tried many methods to modify policy head and value head.Now this is the strongest one.I‘m not very clear why it works well.

[l1t1]

interesting, can you attatch the file, or upload pan.baidu

[l1t1]

can you play a bot on http://www.yss-aya.com/cgos/9x9/standings.html ?

[l1t1]

or release here, including the windows exe
btw, pls use 7z's gz format for weights

[alreadydone]

Great work, and let me congratulate on its release! To make it clear, the code modifies only the three lines in the weight file whose lengths depend on the board size:
m_ip_pol_w, size 362x2x361 (outermost dimension first)
m_ip_pol_b, size 362
m_ip1_val_w, size 256x361
using various methods of interpolation; the best method was used to produce the provided weights. I came up with the idea of deleting some rows/columns from those three lines, but thought that the resulting weights would be very weak. It's @hzyhhzy's diligent coding, testing, tuning and invention of new methods that established that the strength loss is tolerable; the converted 13x13 weights even outperformed weights trained for many days. (Thanks also to @pangafu and @sbf2000 for constructive suggestions.)
The converted weights, the corresponding leelaz engines, and a version of Lizzie that supports all these odd board sizes, are also available as a package from https://pan.baidu.com/s/1tutkE8PfuY-Jzi9hBn50dQ

[l1t1]

the compress diff is about 10%, but the gz format can be used directly

D:\leelaz -w 151s.txt.gz
Using 2 thread(s).
RNG seed: 15418827168788825565
Leela Zero 0.16  Copyright (C) 2017-2018  Gian-Carlo Pascutto and contributors
This program comes with ABSOLUTELY NO WARRANTY.
This is free software, and you are welcome to redistribute it
under certain conditions; see the COPYING file for details.

BLAS Core: Haswell
Detecting residual layers...v1...224 channels...16 blocks.

diff

  60,667,018 151s.7z
 189,975,296 151s.txt
  65,600,974 151s.txt.gz

[hzyhhzy]

The modified 9x9 weight is as strong as my 55th trained weight(much weaker than my 158th weight).13x13 is much stronger than the best trained weight.19x19 weight modified from trained 13x13 weight is very weak(as strong as 40th LZ weight.13x13 weight modified from 9x9 weight is weaker(about 10 to 18 kyu).

[l1t1]

maybe you can play against this bot
http://www.yss-aya.com/cgos/9x9/cross/LZ20_128_65_p3000.html

[l1t1]

13x13 bot
http://www.yss-aya.com/cgos/13x13/cross/LZ13_SL_05_p1600.html

[hzyhhzy]

9x9 isn't very strong(maybe as strong as the bot on CGOS).13x13 is very strong.1po can reach the top.

[l1t1]

here is rns 9x9 go programe
https://github.com/zakki/Ray/releases/tag/rn-5.5.1

[l1t1]

I notice the compressed size of each weight are very small except one, indicate that the most content of them are identical, so I wrote following code to find them.

import sys, os, random
def main():
    print('usage fc.py weight1 weight2\n') 
    
    f1 = open(sys.argv[1], 'rt')
    f2 = open(sys.argv[2], 'rt')
    error=0
    while 1:
        v1 = [float(x) for x in f1.readline().split()]
        if len(v1)==0: break
        v2 = [float(x) for x in f2.readline().split()]

        if len(v1)!=len(v2) :
            print(len(v1),'/',len(v2))
        else:
            e = sum((float(v1[i]) - float(v2[i]))**2 for i in range(len(v1)))
            error += e/len(v1)
    print('Weight file difference L2-norm: {}'.format(error**0.5))

    f1.close()
    f2.close()
if __name__ == '__main__': main()

I wonder why the weight are suitable for other size? alphago papers say it can only work with 19x19?
@alreadydone

D:\>c:\python37\python fc15.py 205 15_205.txt
usage fc.py weight1 weight2

261364 / 101700
362 / 226
92416 / 57600
Weight file difference L2-norm: 0.0

[alreadydone]

@l1t1 As I said and you've found out, only three lines are changed (a very small portion of the whole 40b weights). The convolutional filters extract certain features that are common to all board sizes; the problem is just to figure out how to deal with fully-connected layers. Can you point out to me where in which AlphaGo paper it's said that "it can only work with 19x19" and what's the exact statement?

[l1t1]

i didnt find that in paper 1 (https://blog.csdn.net/u013390476/article/details/50925347)

[cryptsport]

The converted weights, the corresponding leelaz engines, and a version of Lizzie that supports all these odd board sizes please upload to https://drive.google.com
can't download from https://pan.baidu.com

[alreadydone]

https://drive.google.com/file/d/1goxpawqozxBdbN50BKpmOhrtgSm6uqqU/view?usp=sharing

All weight files modified from LZ # 205. 9x9 is only slightly stronger than Zen7 and much weaker than trained weights, so it's not included. According to @hzyhhzy, these are superhuman on 13x13, 15x15, and 17x17, but other board sizes are not tested.

Lizzie shortcut keys/combinations:
x : view all shortcut keys/combinations
Ctrl+x : change engine parameters and board size
Ctrl+number : switch engine (still need to change board size with Ctrl+x)
Ctrl+1 : 11x11
Ctrl+2 : 13x13
Ctrl+3 : 15x15
Ctrl+4 : 17x17
Ctrl+5 : 19x19 (original LZ # 205)
Ctrl+6 : 21x21
Ctrl+7 : 23x23
Ctrl+9 : 25x25

[cryptsport]

@alreadydone Thank you very much !

[barrtgt]

Can anyone please share 9x9 binary so we could use these trained nets ihavnoid/leelaz-ninenine#6 for this lizzie?

[alreadydone]

@barrtgt Here you are. leelaz9.zip compiled off latest /next for 9x9

BTW we noticed leelaz compiled for even board sizes seem to be working (though I am aware that the code writing/reading training data is not OK with even size). The method of modifying the weights treat the two dimensions of the board separately and hence we can produce weights for rectangular boards (though transition between sizes with same parity may better preserve strength due to checkerboard pattern in value head), so I wonder if anyone is interested in taking up the challenge to adapt leelaz to support rectangular boards. (Note: NN code already separates width and height; symmetries down from 8 to 4; UI support not ready but should be easy to implement.)

[barrtgt]

Yay! Thank you.

[l1t1]

dose the smaller board size result to shorter think time for a given weight? eg. 205-11 vs 205-19

[l1t1]

at same playouts or visits

[kennyfs]

@hzyhhzy
你可以講解一下改變權重的原理嗎？
Can you explain the principle of changing weights?

[alreadydone]

I wrote a Python script board_resize.py to convert the weights based on my understanding of the methods according to discussions with @hzyhhzy and others. It's completely written in terms of matrix operations with numpy and scipy.sparse. I implemented two methods, called iterative method and rescaling method respectively. Let me explicate the details, in the hope of inspiring further work and progress.

The iterative method

As an example, to produce a 4x4 weight from a 5x5 weight, we use the following matrix (single_step_mat in the .py):

4/5	1/5	0	0	0
0	3/5	2/5	0	0
0	0	2/5	3/5	0
0	0	0	1/5	4/5

(plz ignore the boldface and coloring github put on my matrix (entered as a table))

The matrix has 4 rows and 5 columns which correspond to 4x4 and 5x5 respectively. For example, 4/5,1/5,0,0,0 means that the 1st line (line = row or column, 1st line = edge of board) of the 4x4 weight is 4/5 times the 1st line plus 1/5 times the 2nd line of the 5x5 weight.

You can see that the 1st/last line gets a large contribution from the 1st/last line, so the peculiarity of the edge/corner is preserved. As we approach the center of the board, the weighting of the two adjacent lines approaches half-half.

You could print out matrices (single_step_mat, iter_mat, rescale_mat) with different arguments as at the end of the .py file.

To go from 5x5 to 3x3, we just first go from 5x5 to 4x4, then from 4x4 to 3x3, which corresponds to matrix multiplication/composition.

This matrix is called of stride one, having many adjacent entries. Since the 256x361 weights in the value head exhibit checkerboard patterns (alternate between big and small values), stride-one can't be expected to work well on those weights, and we use a stride-two matrix to capture the even/odd parity. The following is the stride-two matrix to go from 5x5 to 3x3:

3/4	0	1/4	0	0
0	2/4	0	2/4	0
0	0	1/4	0	3/4

You can see the 1st and 3rd line (where 1 and 3 have the same parity) of the 5x5 weight goes into the 1st line of the 3x3 weight.

To enlarge the board size, for example from 4x4 to 5x5, I use the following:

1	0	0	0
3/4	1/4	0	0
0	2/4	2/4	0
0	0	1/4	3/4
0	0	0	1

and from 4x4 to 6x6, with stride 2:

1	0	0	0
0	1	0	0
1/3	0	2/3	0
0	2/3	0	1/3
0	0	1	0
0	0	0	1

the rescaling method

The idea is to rescale an image of mxm squares(pixels) to fit in one of nxn squares. Then, for each square x in the mxm image and each square y in the nxn image, calculate the portion of x that lies in y; then the weight at y will contribute that portion to the weight at x. Here is it in the matrix form (5x5->3x3, stride 1):

3/5	2/5	0	0	0
0	1/5	3/5	1/5	0
0	0	0	2/5	3/5

(note not only each row adds up to 1 (as we're doing weighted averaging) but also each column sums up to 3/5)

and from 5x5 to 7x7 with stride 2 (splits into 3x3->4x4 for odd lines and 2x2->3x3 for even lines):

3/3	0	0	0	0
0	2/2	0	0	0
1/3	0	2/3	0	0
0	1/2	0	1/2	0
0	0	2/3	0	1/3
0	0	0	2/2	0
0	0	0	0	3/3

The rescale_mat function can treat "squares" on the edge as a 1x0.5 rectangle and squares in the corner as a 0.5x0.5 square, if edgewt is set to 0.5.

---

By default, the iterative method is used for the value head weights (with stride 2) and the policy head biases (with stride 1). For the policy head weights, things are more complicated; I group together connections from i to j according to the value of (i-j) (so for nxn board there are (2n-1) groups), and apply the iterative method to each group. This is what single_step_fc_mat and iter_fc_mat are for. We could also apply the rescaling method to each group (with any stride if shrinking and stride 1 if enlarging), but I haven't implemented it.

The rescaling method is implemented but not used by default; you may try the one suggested for val_w_mat in the .py.

Finally, the value head weights need to be renormalized because the number of inputs change with board size; I use the quotient of the Euclidean lengths of the weight vectors as the factor (multiplier) for each filter. (Same for the policy head weights, but there the factor seems always close to 1.)

However, even after these renormalization, we still observe that the policy of pass goes too low when board is shrinked and goes too high when it's enlarged. Also, for Minigo v15 weights (at least for # 990), the initial black winrate seems to increase as the board size is shrinked, from 60%+ for 17x17 to 90%+ for 13x13. I haven't got the chance to do a thorough investigation to figure out the reason and improve the methods. Maybe these problems can't be solved with modifying the weights without training with self-play games.

My implementation seems to produce weights of similar strength as those produced by hzyhhzy's cpp.
The current CGOS top, ELFv1_13_p1600 (so far only lost games on time), was produced with my script, and mgLZ_204_v4_1po/10po/100po were produced with the cpp program.

[alreadydone]

Since this approach works, albeit not perfectly, it seems very promising to freeze all weights in the residual blocks and train the policy/value heads only, starting from a converted model for self-play generation. I found that this is typically done with tf.stop_gradient or setting trainable=False in tf.Variable (I think the former is preferable), but I don't immediately see how I'd modify tfprocess.py. @Ttl: You mentioned in #1030 (comment) (March 2018) that you have the code already written (for freezing some layers); could you please share it? Thanks in advance.

[Ttl]

Set trainable=False for all variables you don't want to train and don't forget to also make batch normalization non-trainable. Batch norm should have training as before, but add trainable=False.

[alreadydone]

@Ttl Thanks! will look into it.

---

BTW I forgot to say how to use the script:
python3 board_resize.py.txt 〈original weight file name〉 〈target board size〉

[Vandertic]

However, even after these renormalization, we still observe that the policy of pass goes too low when board is shrinked and goes too high when it's enlarged.

@alreadydone you could maybe tune the renormalization so that the policy of pass is kept constant.

[wonderingabout]

9x9 isn't very strong(maybe as strong as the bot on CGOS).13x13 is very strong.1po can reach the top.

@hzyhhzy

most likely because 9x9 is too small as compared to the original 19x19, and the 19x19 strategies do not work well to win on 9x9, and same for too big boards as compared to original 19x19 (strategies to win 50x50 are for sure very different than 19x19 for example)
a 17x17 or 21x21 are probably close in strength percentage as compared to the original 19x19 (maybe 60-70% of the original strength)
but 13x13 or 25x25 are probably much weaker in strength percentage as compared to the original 19x19 (maybe 30% or less of the original strength)

just a hypothesis, but the strength curve ratio as compared to the original 19x19 is probably a bell-like curve, with its peak being 19x19 in the midddle
(i.e : boardsizes too small or too big as compared to original boardsize result in too big strength loss relatively)

this would be because the strategies to win strongest go change when boardsize is different
and also because the conversion should not be lossless (compare to weight trained on 13x13 natively for example, even with less training time)

something like this :

picture credit (totally uninteresting here) https://www.quora.com/What-percentage-of-diseases-have-unknown-pathophysiology

[seopsx]

Use the option --cpu-only @criptopark

[l1t1]

try play against rn5.5.1
https://github.com/zakki/Ray/releases/tag/rn-5.5.1

[cryptsport]

@seopsx Thank you very much !

[cryptsport]

Has anyone tried 25x25 networks? elf v1 and v2 converted by ChangeBoardSizeOfWeight.cpp and board_resize.py never won against 25_205. (time parity)
@hzyhhzy, @alreadydone
maybe your programs are focused on 40 block network?

[alreadydone]

Make sure you change blocks=15 to 20 in ChangeBoardSizeOfWeight.cpp, but my script doesn't care about the number of blocks. My script doesn't have any tuned parameters but hzyhhzy tuned a lot by experiments, so that's probably why his converted 205 works better. However, 0% winrate would be surprising. How many games have you tested?

[jillybob]

What are the strongest 9x9, 13x13 networks produced so far using this method? Can I have links please?

[cryptsport]

@alreadydone
yes, block = 20. Yes, there are few games, but the transformed elf plays very strange (some moves are very silly).
  and Eval (even at the beginning of the game), transformed elf, 25x25, black, first moves:
1 29.27%
3 21.26%
5 5.53%
7 22.33%
9 5.47%
11 3.37%
13 3.49%
15 3.67%
another party:
1 45.16%
3 38.15%
5 9.34%
7 3.10%
9 4.12%
11 2.00%
13 3.14%
15 2.48%
(original elf 19x19, black), first moves:
1 48.76%
3 48.49%
5 49.86%
7 50.84%
9 50.33%
11 50.67%
13 50.12%
15 50.30%

[cryptsport]

@jillybob
of the tested 13x13 networks, probably the strongest elfv1 (converted by board_resize.py): https://drive.google.com/file/d/1vPdzfqOvu0YPJbU25PYgZYMAcFH4O3-L/view?usp=sharing
but as far as I know, only a few 13x13 converted networks were tested

[Marcin1960]

"of the tested 13x13 networks, probably the strongest elfv1 (converted by board_resize.py)"

Running on KGS as LeelaZeroT. Enjoy :)

[cryptsport]

can someone make leelaz37, leelaz49 and GoGui up to 52x52 (for the game and gogui-twogtp)?
interesting: https://senseis.xmp.net/?LargeBoards

[jillybob]

@criptopark @Marcin1960 Can you upload the Windows Leelaz 13x13 binary please?

[cryptsport]

@jillybob link dated 02.22.19 published @alreadydone working
https://drive.google.com/file/d/1goxpawqozxBdbN50BKpmOhrtgSm6uqqU/view?usp=sharing

[cryptsport]

@hzyhhzy thanks for the great idea and program!
ChangeBoardSize2 improved version, if you use it for odd boards?
I made elfv2_25 by first and second versions, the files received are completely match up
except for the section ~ 9kb which consists of zero values (obtained using the second version). then I noticed that in the converted Files contain many sections with zero values of ~ 2 kb each, but there are no large sections of zeros in the original elfv2. this is normal? or received files
containing these plots of zeros are defective?

[hzyhhzy]

Yes,only has differences when even board sizes.Zeros depend on algorithm,not bug.But I found that 25x25 or larger boards(very obvious on 37x37 or larger)tend to avoid playing on the center of board.It would fill the corner and side rather than playing on the center of board.

[cryptsport]

@hzyhhzy thanks for the answer! 37x37 - very interesting! Where can I download leelaz37?

[trunterzx]

#2240 (comment)

@alreadydone
Do you think this script board_resize.py could be used by a layman who doesn't know python?
I would like to try with the newer weights if possible...

[cryptsport]

@trunterzx I don't know python. To run the script you need a python + numpy or anaconda. The startup example is: python board_resize.py.txt elf.txt 25, where elf.txt is the original network, 25 is the resulting size. If you have any questions, write.
and you can compile leelaz 37x37?

[poptangtwe]

@lightvector

[cryptsport]

several matches 25x25, nets received by ChangeBoardSizeOfWeight.cpp, 10sec/move, cpuonly, gogui-twogtp:

LM 192x15 GX89 - LZ 40x256 #205 25:15
LZ 192x15 f438268e - LZ 40x256 #205 18:22
elf v2 256x20 - LZ 40x256 #205 17:23, black elf all parties (11) won because of the ladder
converted minigo(25x25) 000930-goliath and 000990-cormorant do not work in gogui and sabaki

[alreadydone]

KataGo is trained to adapt to different board sizes, and now you can get Windows binary here: KataGo GTP engine and networks (up to 37x37, arbitrary komi, handicap play). Instructions.txt included.
lightvector/KataGo#2 (comment)

[cryptsport]

several matches 25x25, nets received by board_resize.py.txt vs
LZ 40x256 #205 by ChangeBoardSizeOfWeight.cpp, 10sec/move, cpuonly, gogui-twogtp:

LM 192x15 GX89 - LZ 40x256 #205 13:27
LZ 192x15 f438268e - LZ 40x256 #205 5:35
elf v2 256x20 - LZ 40x256 #205 12:28
converted minigo(25x25) 000990-cormorant works, did not test
and LM 192x15 GX89(by ChangeBoardSizeOfWeight.cpp) - LM 192x15 GX89(by board_resize.py.txt) 37:3 (White 20:0)

[mc-mong]

minigo V17 .. ChangeBoardSizeOfWeight.cpp

Is it supportable?

[katagoni]

https://drive.google.com/open?id=1I7LtMcuwyLUZDNjV5VL9eUwTDdyA2nIT

Download your file and wanna chage 19x19 in sabaki you configruation 9x9 how to change

[katagoni]

Here are some weights from 9x9 to 25x25.(converted from #205(85a93684) lz weight),

https://drive.google.com/open?id=1jQjCbK1xWGBglD402WpBMyeEieDG-h83

and a very rough CPP code.

https://drive.google.com/open?id=1bgkVBeCIwTgA88DG07sQE8oXHUdDuqt7

I tried many methods to modify policy head and value head.Now this is the strongest one.I‘m not very clear why it works well.

https://drive.google.com/open?id=1I7LtMcuwyLUZDNjV5VL9eUwTDdyA2nIT

Download your file and wanna chage 19x19 in sabaki you configruation 9x9 how to change

[fantianwen]

A bot “Golaxy” from Tencent, a China-based company, has been seen to run on multiple board sizes, but we are not aware of anywhere they have published their methods.

They are using Transfer Learning for various board sizes. can be checked here:
https://www.19x19.com/intro?from=%2Findex&to=%2Findex
(Chinese version only)

[mushoku-ningen]

im a unfortunately mac user. is there any way for me to play 13x13 igo against leela-zero. i downloaded above 13x13 weight file and added it to my engine. but i cant step out of the "engine is loading" screen

[cryptsport]

@mushoku-ningen leela-zero works only with a 19x19 network. for a 13x13 network, a compiled file is needed that works with a 13x13 network.
for windows for networks 11x11 - 25x25 here: https://drive.google.com/file/d/1goxpawqozxBdbN50BKpmOhrtgSm6uqqU/view?usp=sharing
ps KataGo works with any boards, but here up to 29x29: https://github.com/lightvector/KataGo/releases/tag/v1.3.5%2Bbs29

[mushoku-ningen]

@cryptpark thanks! really helpful!