Create and collect pgn of the test games / biased workers

[Matthies]

More a feature request than an issue...

It would be great if cutechess would create pgn files of test games and workers would collect, zip and send them to server for download.

There are some really strange results in some tests and I would like to run some evaluation on the pgns to see if its just statistical noise or some real problem of workers.

Example: http://chess.grantnet.us/viewTest/4668/ should be (almost) completely neutral but in this moment:
Andrews worker ist at +1044 -1159 =2725
My worker ist at +772 -647 =1717

[Matthies]

Meanwhile I made some investigation on my biased worker. It turned out that the binary of the dev version was ~10% faster than the master binary (although both are practically identical). A copy of the slow master binary was as fast as the dev. Also a test with Linux perf tool showed this:
Slow binary: 271.841.763.953 cycles:u
Fast copy: 250.872.200.128 cycles:u
After a reboot of the computer everything was completely different. Now master and dev seem almost equal in speed. One was faster than before reboot, one was slower.
All this on a Ryzen3700x running Ubuntu 19.10. Speed tests described here ran with 1 thread.
Very weird all in all and reminds me of the strange results you had with your ep ^ 8 simplification.

[Matthies]

I have removed -flto now from make which seemed to influence speed bias but today I got another very bad test on my worker. Look at the nps difference of the following bench runs. The engines only differ in some pawn eval params:

andreas@andreas-MS-7B86:~/github/OpenBench/Client$ ./Engines/6748bd88b7912d7306caba1af7eb33bd855d580b -bench -depth 23 > /dev/null 


Benchmark results for RubiChess 1.7-dev 20200108200410 (Build Mar  7 2020 12:19:45 commit 09cc9d1):
System: some Linux box
=============================================================================================================
Bench #   1 ( Startposition / 14): o  e2e4      20 cp  23 ply   3.105649 sec.    5521037 nodes    1777740 nps
Bench #   2 (   Lasker Test / 28): o  a1b2     144 cp  23 ply   0.020451 sec.      49383 nodes    2414729 nps
Bench #   3 (        IQ4 63 / 14): o  f3e5     533 cp  23 ply   0.574013 sec.    1170261 nodes    2038735 nps
Bench #   4 (    Wacnew 167 / 14): o  f2g2   32500 cp  23 ply   0.072005 sec.     134351 nodes    1865857 nps
Bench #   5 (    Wacnew 212 / 14): o  h8g7   32500 cp  23 ply   0.060596 sec.     107477 nodes    1773674 nps
Bench #   6 (      Carlos 6 / 13): o  b3e6     554 cp  23 ply   4.589790 sec.    6726402 nodes    1465513 nps
Bench #   7 (   Arasan19 83 / 14): o  a7a5     161 cp  23 ply   1.623438 sec.    3222158 nodes    1984774 nps
Bench #   8 (  Arasan19 192 / 13): o  f1f7       2 cp  23 ply   2.092676 sec.    3363192 nodes    1607125 nps
Bench #   9 (     BT2630 12 / 15): o  e6f5    1161 cp  23 ply  10.182835 sec.   22047133 nodes    2165127 nps
Bench #  10 (       IQ4 116 / 14): o  e4f2     293 cp  23 ply   0.856769 sec.    1554424 nodes    1814286 nps
Bench #  11 (  Arasan12 114 / 15): o  f1e2      57 cp  23 ply   2.752254 sec.    4254207 nodes    1545717 nps
Bench #  12 (  Arasan12 140 / 15): o  f3g3     281 cp  23 ply   7.006227 sec.   10714192 nodes    1529238 nps
Bench #  13 (  Arasan12 137 / 14): o  g5g6     311 cp  23 ply   2.460275 sec.    3707793 nodes    1507064 nps
=============================================================================================================
Overall:                                                       35.396976 sec.   62572010 nodes    1767721 nps
andreas@andreas-MS-7B86:~/github/OpenBench/Client$ ./Engines/bf4f97bc4bfc256547c6f9a5092d8aa9d0f3e77d -bench -depth 23 > /dev/null 


Benchmark results for RubiChess 1.7-dev 20200108200410 (Build Mar  7 2020 16:21:28 commit 09cc9d1):
System: some Linux box
=============================================================================================================
Bench #   1 ( Startposition / 14): o  d2d4      34 cp  23 ply   3.872284 sec.    8541721 nodes    2205860 nps
Bench #   2 (   Lasker Test / 28): o  a1b2     122 cp  23 ply   0.020445 sec.      55882 nodes    2733255 nps
Bench #   3 (        IQ4 63 / 14): o  f3e5     563 cp  23 ply   1.401215 sec.    3818294 nodes    2724988 nps
Bench #   4 (    Wacnew 167 / 14): o  f2g2   32500 cp  23 ply   0.051557 sec.     123729 nodes    2399860 nps
Bench #   5 (    Wacnew 212 / 14): o  h8g7   32500 cp  23 ply   0.050398 sec.     100351 nodes    1991174 nps
Bench #   6 (      Carlos 6 / 13): o  b3e6     496 cp  23 ply   2.732808 sec.    5031567 nodes    1841171 nps
Bench #   7 (   Arasan19 83 / 14): o  a7a5     196 cp  23 ply   1.672060 sec.    3942099 nodes    2357630 nps
Bench #   8 (  Arasan19 192 / 13): o  f1f7      93 cp  23 ply   2.056769 sec.    4059708 nodes    1973827 nps
Bench #   9 (     BT2630 12 / 15): o  e6f5    1076 cp  23 ply   3.760445 sec.    9236138 nodes    2456129 nps
Bench #  10 (       IQ4 116 / 14): o  e4f2     313 cp  23 ply   0.786478 sec.    1819418 nodes    2313373 nps
Bench #  11 (  Arasan12 114 / 15): o  a4b6     244 cp  23 ply   3.115057 sec.    6052912 nodes    1943114 nps
Bench #  12 (  Arasan12 140 / 15): o  e4d6     138 cp  23 ply   3.881429 sec.    7505968 nodes    1933815 nps
Bench #  13 (  Arasan12 137 / 14): o  g5g6     253 cp  23 ply   2.540481 sec.    4782252 nodes    1882419 nps
=============================================================================================================
Overall:                                                       25.941425 sec.   55070039 nodes    2122860 nps

Still not clear to me what's going wrong. Is it the compilation, is it Linux, is it Ryzon?
Edit: Not the compilations fault. A copy of the 6748bd88b7912d7306caba1af7eb33bd855d580b engine file runs as fast as it should (2152372 nps).

[AndyGrant]

I don't have a clue. It could be the compiler, it could be the OS, it could be the platform.

But this issue is really not OpenBench specific, so I would suggest posing it on Talkchess to get some more eyes on it. What compiler are you using? and what version? I personally don't trust any GCC after 4.8.

[Matthies]

Well, probably not the compiler. I just have to copy the binary to make it run fast. I also tested if some Linux process has a handle on the slow engine binary file. Nope.
I posted here cause it has big influence on the trustworthiness of Openbench tests. I saw several very biased workers in the past, not only mine.
I will make a post in talkchess later...

[AndyGrant]

Is this something that can be reproduced? For this particular branch, can you, repeated, do the following: Clone the branch. Build the branch. Run it and see that it is slow. Run it again and see that it is slow. Copy the binary elsewhere. Run it and see it is not slow.

If that is repeatable, does it hold true for GCC4.8? GCC5? GCC6? GCC7? GCC8? CLANG?

[Matthies]

It was the master branch that ran slow. This same master binary ran fast when it was compiled by Openbench worker. Well, fast enough to score positive against my first attempt of pawntune: http://chess.grantnet.us/test/4853/
Later that day I started Linux on my computer again (was doing some tests under Windows in the meantime) and ran Openbench client again to run the second pawntune test. And now the (reused) master binary ran slow.
I will do some tests with several reboots today and see if this binary runs fast or slow.

One thing I didn't mention so far and I don't know if its related: When I started Openbench Client the first time on this biased test, all the first games ended in a "loss on time". See http://chess.grantnet.us/test/4856/
So I stopped the Client and started it again and everythings seemed fine. Until I realized that the test scores +20 Elo...
Edit: The "loss on time" problem is clear to me now. Running Windows before which uses local time instead of UTC triggers a time correction in Linux. And if that happens when the worker is already running engines get confused and lose on time.
I have done 5 reboots now and even one into Windows and the master binary always ran fast (>2.1mnps). So no reproduction so far. Will do more tests later...

[AndyGrant]

Added better error reporting to the server: 0c8ee1c
Added a client-side option to save PGNs locally: 5e7aead

I believe this is the most I can do to address this, as I do not have the means to store the PGNs on the server, nor do I want to mandate that Workers upload PGNs. I will run my workers with PGN storage on, so in the event there is an issue, at a minimum, I have the PGN for the game that had an issue.