Cutechess falsely adjudicates a game via an Illegal move (FRC)

[AndyGrant]

$ ./cutechess-dev --version
cutechess-cli 1.2.0
Using Qt version 5.12.8

I'm using the official release provided here. I am seeing one illegal move in about every 80,000 games played. although sometimes it occurs much more frequently. Note that in both these examples we have black castling when there are discovered checks on the back rank. The following is the cutechess command passed:

./cutechess-dev -variant fischerandom -recover -resign movecount=2 score=400 -draw movenumber=40 movecount=4 score=20 -concurrency 32 -games 1125000 -engine cmd=Ethereal proto=uci tc=1.00+0.01 option.Hash=2 option.Threads=1 name=Ethereal -engine cmd=Ethereal proto=uci tc=1.00+0.01 option.Hash=2 option.Threads=1 name=Ethereal -openings file=Books/6plies_FRC.pgn format=pgn order=random -pgnout EtherealFRC_3.pgn -tb /home/andrew/Desktop/Syzygy/

Here is an example of the game which was falsely flagged. https://lichess.org/B9XZx7TH

[Event "?"]
[Site "?"]
[Date "2020.09.03"]
[Round "1"]
[White "Ethereal"]
[Black "Ethereal"]
[Result "1-0"]
[FEN "bnr1nqkr/pp1p1ppp/4pb2/2p5/4P3/1PN5/P1PP1PPP/B1RBNQKR w KQkq - 2 7"]
[GameDuration "00:00:01"]
[GameEndTime "2020-09-03T04:22:20.453 EDT"]
[GameStartTime "2020-09-03T04:22:18.461 EDT"]
[PlyCount "45"]
[SetUp "1"]
[Termination "illegal move"]
[TimeControl "1+0.01"]
[Variant "fischerandom"]

1. Nb5 {+0.04/11 0.063s} Nc6 {-0.04/12 0.095s} 2. e5 {-0.11/11 0.028s}
Nxe5 {+0.13/12 0.027s} 3. Nxa7 {-0.29/12 0.093s} Rd8 {+0.32/13 0.098s}
4. Nb5 {-0.46/13 0.20s} b6 {+0.40/12 0.047s} 5. h3 {-0.42/12 0.021s}
Bc6 {+0.45/12 0.099s} 6. c4 {-0.54/10 0.045s} Ng6 {+0.52/11 0.022s}
7. Bf3 {-0.75/10 0.070s} Bxa1 {+0.66/12 0.066s} 8. Rxa1 {-0.90/12 0.096s}
h5 {+0.79/11 0.037s} 9. a4 {-0.55/12 0.044s} Nf6 {+0.66/11 0.048s}
10. Qe2 {-0.53/11 0.019s} Bxb5 {+0.75/11 0.043s} 11. cxb5 {-0.78/11 0.041s}
e5 {+0.85/12 0.021s} 12. d3 {-1.02/12 0.052s} d5 {+0.83/12 0.029s}
13. a5 {-0.91/11 0.017s} bxa5 {+0.90/12 0.033s} 14. Qd2 {-1.05/11 0.018s}
Qe7 {+0.93/11 0.039s} 15. Rxa5 {-1.10/11 0.077s} e4 {+1.03/12 0.019s}
16. Bd1 {-0.82/10 0.018s} exd3 {+1.74/12 0.029s} 17. Nxd3 {-0.25/11 0.018s}
c4 {+1.67/12 0.023s} 18. Nb4 {-0.01/11 0.010s} Ne4 {-0.54/11 0.039s}
19. Qe1 {+0.47/11 0.012s} c3 {+0.19/13 0.068s} 20. Nc6 {-0.51/10 0.009s}
Qg5 {-0.59/12 0.025s} 21. Nxd8 {+1.41/10 0.010s} Nf4 {-0.17/13 0.044s}
22. Bf3 {+1.70/10 0.010s} c2 {-0.59/9 0.019s}
23. Ra8 {+0.35/8 0.010s, Black makes an illegal move: g8h8} 1-0

Here is a second example: https://lichess.org/cxkWGzmC#12

[Event "?"]
[Site "?"]
[Date "2020.09.03"]
[Round "1"]
[White "Ethereal"]
[Black "Ethereal"]
[Result "1-0"]
[FEN "q1bbnrkr/pppnppp1/7p/3p4/8/1P2P2P/P1PP1PP1/QNBBNRKR w KQkq - 1 7"]
[GameDuration "00:00:02"]
[GameEndTime "2020-09-03T04:18:59.700 EDT"]
[GameStartTime "2020-09-03T04:18:57.593 EDT"]
[PlyCount "55"]
[SetUp "1"]
[Termination "illegal move"]
[TimeControl "1+0.01"]
[Variant "fischerandom"]

1. c4 {0.00/11 0.060s} dxc4 {-0.03/12 0.065s} 2. bxc4 {+0.10/13 0.11s}
b6 {-0.07/11 0.031s} 3. Bb2 {+0.12/13 0.058s} e6 {+0.04/12 0.043s}
4. d3 {+0.13/13 0.040s} Bf6 {-0.18/12 0.072s} 5. e4 {+0.06/12 0.040s}
Bxb2 {-0.24/12 0.032s} 6. Qxb2 {+0.07/12 0.038s} Bb7 {-0.08/12 0.026s}
7. Nd2 {+0.09/12 0.036s} Nd6 {-0.12/12 0.059s} 8. f4 {+0.06/11 0.052s}
c5 {-0.05/12 0.033s} 9. Nef3 {+0.22/11 0.041s} Rd8 {-0.09/12 0.020s}
10. Re1 {+0.04/11 0.040s} Nc8 {-0.13/11 0.044s} 11. O-O {-0.04/11 0.056s}
Qb8 {+0.10/12 0.045s} 12. Ba4 {-0.06/11 0.038s} Nf8 {+0.53/12 0.031s}
13. f5 {-0.23/11 0.043s} exf5 {-0.14/11 0.028s} 14. exf5 {+0.22/12 0.049s}
f6 {-0.04/12 0.054s} 15. Qc3 {-0.17/11 0.025s} Nd6 {+0.29/12 0.026s}
16. Ne4 {+0.20/12 0.037s} Nh7 {+0.21/12 0.024s} 17. Nh4 {+0.18/13 0.044s}
Nxe4 {-0.01/13 0.083s} 18. dxe4 {+0.53/12 0.032s} Ng5 {-0.54/13 0.045s}
19. Ng6 {+0.58/13 0.099s} Bxe4 {0.00/12 0.15s} 20. h4 {+2.51/11 0.015s}
Rd3 {-0.53/11 0.032s} 21. Qc2 {+1.61/9 0.010s} Rd4 {-0.92/10 0.016s}
22. Qe2 {+0.60/10 0.011s} Nh3+ {-0.17/9 0.010s} 23. Kh1 {+1.30/9 0.009s}
Bd3 {-1.07/9 0.014s} 24. Qf3 {+3.59/9 0.011s} Bxf1 {-2.20/10 0.012s}
25. Re8+ {+1.78/10 0.009s} Qxe8 {-2.56/10 0.004s} 26. Bxe8 {+2.17/11 0.017s}
Nf4 {-2.29/9 0.009s} 27. Nxf4 {+2.55/9 0.004s} Bxc4 {-3.22/9 0.011s}
28. Qa8 {+3.96/11 0.010s, Black makes an illegal move: g8h8} 1-0

[AndyGrant]

3rd case, this time not relating to FRC explicitly: https://lichess.org/Dvplq83n#56

[Event "?"]
[Site "?"]
[Date "2020.09.03"]
[Round "1"]
[White "Ethereal"]
[Black "Ethereal"]
[Result "0-1"]
[FEN "rkrnq1bb/1ppppp1p/p5n1/6p1/P3P3/3P4/1PP2PPP/RKRNQNBB w KQkq - 0 7"]
[GameDuration "00:00:01"]
[GameEndTime "2020-09-03T04:54:20.902 EDT"]
[GameStartTime "2020-09-03T04:54:19.010 EDT"]
[PlyCount "44"]
[SetUp "1"]
[Termination "illegal move"]
[TimeControl "1+0.01"]
[Variant "fischerandom"]

1. Qd2 {+0.26/12 0.051s} f6 {-0.37/12 0.11s} 2. g3 {+0.31/11 0.061s}
Qf7 {-0.51/12 0.043s} 3. Nfe3 {+0.45/12 0.067s} e6 {-0.08/13 0.057s}
4. Nc4 {+0.19/11 0.048s} Nc6 {-0.05/12 0.030s} 5. a5 {-0.18/12 0.10s}
Nge7 {+0.31/12 0.048s} 6. f4 {-0.44/12 0.067s} d5 {+0.29/11 0.030s}
7. Na3 {-0.34/12 0.036s} gxf4 {+0.40/12 0.046s} 8. gxf4 {-0.36/12 0.023s}
Qh5 {+0.28/12 0.025s} 9. Bc5 {-0.30/11 0.034s} Re8 {+0.48/11 0.054s}
10. Nc3 {-0.03/12 0.12s} O-O-O {+0.21/12 0.067s} 11. Na4 {-0.22/10 0.019s}
Kb8 {+0.05/11 0.033s} 12. Rg1 {-0.30/10 0.026s} Bf7 {+0.30/10 0.027s}
13. Rg3 {-0.11/11 0.028s} Qh6 {+0.57/11 0.031s} 14. Qf2 {-0.77/10 0.040s}
dxe4 {+0.91/11 0.032s} 15. Bxe4 {-0.68/12 0.060s} f5 {+1.03/12 0.028s}
16. Bxc6 {-1.25/12 0.025s} Nxc6 {+1.34/12 0.025s} 17. Nc4 {-1.06/14 0.052s}
e5 {+1.61/12 0.023s} 18. Nxe5 {-1.27/13 0.025s} Nxe5 {+1.83/12 0.046s}
19. fxe5 {-0.98/13 0.043s} Bxe5 {+0.86/12 0.085s} 20. Rf3 {-1.41/11 0.018s}
Qxh2 {+2.14/12 0.029s} 21. Rxf5 {-0.72/9 0.009s} Bg3 {+5.11/12 0.017s}
22. Qxh2 {+3.57/9 0.011s}
Re1+ {+M1/127 0.005s, White makes an illegal move: f2h2} 0-1

Re1+ delivers checkmate. Ethereal should not be asked to respond to a mating move -- that behavior us is undefined. Cutechess fails to adjudicate this game after Re1+. Does cutechess think Ethereal can CASTLE OUT OF check?

[alwey]

Thank you for reporting these problems. I can confirm all three cases. These are good examples. It is strange that this has not been discovered before.

Cutechess does not correctly handle Rook (or Queen) attacks on the King that are blocked by another attacking piece (Knight in example 1 and Bishop in example 2). Cutechess should allow castling in these cases.

Indeed, cutechess does not always prevent a player from castling out of check. It assumes it is fine for the King to hide behind the Rook after castling (example 3). In fact it is checkmate.

I will push a patch soon. This patch will need a lot of testing in order to avoid a regression.

[AndyGrant]

I am unable to (trivially) build QT applications on Linux. If you could provide me with an updated binary after the patch, I still have another 1.5million FRC games to play out for what I need. I know you are referring to speed in regards to regression of course, but I can do what I can to assist.

[alwey]

Thank you, much appreciated!

[alwey]

@AndyGrant
cutechess-cli-1.2.0-a58959a.zip

This has been built under Ubuntu 18.04.5 using the older Qt 5.9.5. I do not think this is a problem.

I hope this helps.

[AndyGrant]

Thanks -- more games running now. I'll grep for illegal moves every so often, but I figure you resolved the issue completely just via the test positions.

You are welcome to close this issue whenever you see fit.

[alwey]

I cannot close the issue, because I am no maintainer. Please leave this issue open. It will be closed automatically if the patch or an improved version of it gets accepted by the maintainers.

[vondele]

Is this issue specific to FRC or is this also affecting non-variant (standard) chess? I tried to understand the patch but wasn't sure.

[ppigazzini]

I am unable to (trivially) build QT applications on Linux.

@AndyGrant great catch :)
I updated the scripts to build cutechess-cli 1.2.0 on Ubuntu 14.04 for fishtest, easy but it takes some time, see https://github.com/glinscott/fishtest/wiki/Build-cutechess-with-Qt5-static

I use a lxc/lxd container on Ubuntu 20.04, see the last script.

[alwey]

AFAICS the reported problems cannot occur in standard chess as played from a standard starting position. The problems shown are connected with Rook style attacks on a King staying in place or moving towards the attacking Rook when castling. This can happen in FRC but not in FIDE chess. Other chess variants are affected, too. I found that a problem in Grid Chess has been resolved by the patch.

The patch will however affect standard chess if buggy. It is in the execution path for all Western chess variants.

[Nordlandia]

I tried fen: Q3B1kr/p5p1/1p3p1p/2p2P2/2br1N1P/8/P5P1/7K b k - 1 34 in the GUI and black is not allowed to castle strangely enough. Not discovered before now.

@vondele it seems to be related to FRC solely.

[alwey]

@Nordlandia : This is @AndyGrant's second example. It cannot occur in standard chess. I have included this example into the patch as a test case for FRC, see version ce2ffb7 above.

[AndyGrant]

I've played about 2 million FRC games with your new binary, and have not run into any issues (as expected!). Thanks for the fix, and quickly. It would have taken me far too long to figure out where the relevant code was.