Wrong mate scores ?

[MJZ1977]

While debugging some SF games, I found a strange behavior with SF returning a mate score while it is a draw by repetition. Here an example

Fen : 1q3rk1/2R3p1/5p2/5P1Q/3ppP1P/4n3/6PK/2R5 b - - 2 41

You can search this position for the moment. After this, you play Nd5, Rxg7, and search again.
Fen after moves: 1q3rk1/6R1/5p2/3n1P1Q/3ppP1P/8/6PK/2R5 b - - 0 42

What is wrong is that at depth 1, SF returns a mate score in the second position. I normal fishtest game, SF will resign because it stops at first depth if there is only 1 possible move.

To correct it for the moment I just avoid exiting at depth 1 even with only 1 possible move. But this is not probably the best way to do it.

Edit: try for correction here. Passed STC [-1,5..0.5] with some ELO win.
https://tests.stockfishchess.org/tests/view/5ed4cd6ef29b40b0fc95a76b

[vondele]

@MJZ1977 this is the uci input for your case:

position fen 1q3rk1/2R3p1/5p2/5P1Q/3ppP1P/4n3/6PK/2R5 b - - 2 41
go depth 20
position fen 1q3rk1/6R1/5p2/3n1P1Q/3ppP1P/8/6PK/2R5 b - - 0 42
go depth 10
ucinewgame

which yields:

Stockfish 300520 64 POPCNT by T. Romstad, M. Costalba, J. Kiiski, G. Linscott
info depth 1 seldepth 1 multipv 1 score cp 16 nodes 33 nps 33000 tbhits 0 time 1 pv b8b2
info depth 2 seldepth 2 multipv 1 score cp -69 nodes 94 nps 94000 tbhits 0 time 1 pv b8b2 c1g1
info depth 3 seldepth 3 multipv 1 score cp 54 nodes 159 nps 159000 tbhits 0 time 1 pv d4d3 h2g1 b8b2
info depth 4 seldepth 5 multipv 1 score cp 87 nodes 270 nps 270000 tbhits 0 time 1 pv d4d3 c7d7 b8f4 h2g1 f4f5
info depth 5 seldepth 7 multipv 1 score cp 259 nodes 431 nps 215500 tbhits 0 time 2 pv e3d5 c1c4 d5c7
info depth 6 seldepth 7 multipv 1 score cp 169 nodes 737 nps 368500 tbhits 0 time 2 pv e3d5 h5e2 e4e3 c7c2 b8f4 g2g3 f4f5
info depth 7 seldepth 8 multipv 1 score cp 291 nodes 945 nps 472500 tbhits 0 time 2 pv e3d5 h5e2 d5c7 e2e4 c7b5
info depth 8 seldepth 10 multipv 1 score cp 235 nodes 2327 nps 775666 tbhits 0 time 3 pv e3d5 h5e2 d5c7 e2c4 g8h7 c4d4 c7b5 d4e4 b5d6
info depth 9 seldepth 13 multipv 1 score cp 0 nodes 3997 nps 999250 tbhits 0 time 4 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 10 seldepth 7 multipv 1 score cp 0 nodes 4442 nps 1110500 tbhits 0 time 4 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 11 seldepth 7 multipv 1 score cp 0 nodes 4971 nps 994200 tbhits 0 time 5 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 12 seldepth 7 multipv 1 score cp 0 nodes 5662 nps 1132400 tbhits 0 time 5 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 13 seldepth 7 multipv 1 score cp 0 nodes 7079 nps 1179833 tbhits 0 time 6 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 14 seldepth 7 multipv 1 score cp 0 nodes 8482 nps 1211714 tbhits 0 time 7 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 15 seldepth 9 multipv 1 score cp 0 nodes 10809 nps 1201000 tbhits 0 time 9 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 16 seldepth 7 multipv 1 score cp 0 nodes 12873 nps 1170272 tbhits 0 time 11 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 17 seldepth 9 multipv 1 score cp 0 nodes 18045 nps 1388076 tbhits 0 time 13 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 18 seldepth 9 multipv 1 score cp 0 nodes 24677 nps 1645133 tbhits 0 time 15 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 19 seldepth 7 multipv 1 score cp 0 nodes 31911 nps 1772833 tbhits 0 time 18 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h5
info depth 20 seldepth 9 multipv 1 score cp 0 nodes 39479 nps 1973950 tbhits 0 time 20 pv e3d5 c7g7 g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
bestmove e3d5 ponder c7g7
info depth 1 seldepth 1 multipv 1 score mate -3 nodes 8 nps 400 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 2 seldepth 3 multipv 1 score mate -3 nodes 19 nps 950 tbhits 0 time 20 pv g8g7 h5g6 g7h8
info depth 3 seldepth 5 multipv 1 score cp 0 nodes 26 nps 1300 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6 g8h8
info depth 4 seldepth 7 multipv 1 score cp 0 nodes 33 nps 1650 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 5 seldepth 7 multipv 1 score cp 0 nodes 40 nps 2000 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 6 seldepth 7 multipv 1 score cp 0 nodes 47 nps 2350 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 7 seldepth 7 multipv 1 score cp 0 nodes 67 nps 3350 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 8 seldepth 7 multipv 1 score cp 0 nodes 99 nps 4950 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 9 seldepth 7 multipv 1 score cp 0 nodes 143 nps 7150 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6
info depth 10 seldepth 7 multipv 1 score cp 0 nodes 247 nps 12350 tbhits 0 time 20 pv g8g7 h5g6 g7h8 g6h6 h8g8 h6g6

however, the engine for a short while thinks it is being mated, not that it has a mate. So I don't think it is a bug, just the proper bestmove / score hasn't been found yet. I don't think this is a bug.

[MJZ1977]

Perhaps my first message was not clear. I have the same search results. But in the cutechess game with fishtest rules, SF "resigns" because it sended a mate score 3 times (even with the right and only move) and cutechess adjudicate it as "0-1" and not as a draw.

[vondele]

OK, I understand now. Basically, you're saying that the scores can be really bad (mate or adjudication scores), leading to incorrectly lost/adjudicated games. That's something to consider.

I've a variant patch here : https://tests.stockfishchess.org/tests/view/5ed562b0f29b40b0fc95a7d0
This will lead to minimal searches (1ms) in case of a single root move.

[mstembera]

@vondele Do we have some idea of the lower bound for the number of ply searched in 1ms?
Edit: Maybe in some unlucky cases w/ the os scheduler we may still only get 1 ply after 1ms? If so the minimum number of ply required to fix this would be more robust.

[vondele]

@mstembera 1ms usually is quite a depth (easily 10 during games), I picked this only because it integrates more cleanly in the time management, without additional condition being introduced (e.g. why depth 4). It is interesting though that also this patch suggests it could be an Elo gainer to search a bit even if there is only 1 move. While that's strange at first sight, maybe it does make sense for all history tables etc, that benefit from a quick update.

Edit: testing the use of a fraction of normal time in case of 1 move only: https://tests.stockfishchess.org/tests/view/5ed5e3eff29b40b0fc95a7fa

[vondele]

so, testing with roughly 10% of the normal time actually passed as an Elo gainer at STC, but not at LTC (https://tests.stockfishchess.org/tests/view/5ed6a4bff29b40b0fc95a875)

So, I'm inclined to go with the 1ms version (https://tests.stockfishchess.org/tests/view/5ed562b0f29b40b0fc95a7d0). @MJZ1977 what do you think about that?

[MJZ1977]

@vondele : Yes, it is OK for me the 1ms version. I think it will correct some random win/losses in fishtest and it is a good thing.

It will be interessant to test the reported bug and see the difference between master and patch after the commit (for example with go search wtime 100 btim 100)

[Matthies]

In my engine in case of an instamove (rootmoves == 1) I don't do any search and just print the evaluation of the move before. Just an idea...

[mstembera]

On line 72 in ucioption.cpp

o["Minimum Thinking Time"] << Option( 0, 0, 5000);

we default minimum thinking time to 0ms. Should we change it to 1ms?

o["Minimum Thinking Time"] << Option( 1, 0, 5000);

[vondele]

@mstembera that option is anyway wrong the way it is currently implemented. It sets the 'optimum_time()', which is still adjusted by various factors, we can easily search less than that. Prime example of course, if there is a single root move ;-). I'd rather remove than fix that option.

[mstembera]

@vondele OK I don't really know the use cases for that option if any. I just wanted to point it out since it seemed related.