SF depth stuck at 22 ply at one endgame position. Possibly infinitive loop.

[PavelChess]

Describe the issue

I played an engine match in Cutechess interface. One game was buggy - SF depth stuck at 22 ply. the final position is endgame without deep tactics.

I compiled version with fix of depth 0 output, and it has same issue.

I also downloaded latest abrok avx2 version (Date: Fri Jan 26 21:15:07 2024 +0100
Timestamp: 1706300107 ), and it also stucks at 22 ply.

This seems to be infinitive loop.

Expected behavior

Stockfish has to increase depth very quickly, because this was endgame without deep tactics, and should quickly get even 30+ ply if requested.

Steps to reproduce

position fen "1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30"
go depth 25

Anything else?

1. Nf3 g6 2. d4 Bg7 3. e3 d6 4. c4 Nf6 5. Nc3 O-O 6. Be2 b6 7. Qc2 Bb7 8. O-O
   Nbd7 9. Rd1 e6 10. h3 d5 11. a4 c5 12. a5 cxd4 13. exd4 dxc4 14. a6 Bd5 15. Ne5
   b5 16. Nxb5 Qb8 17. Nxd7 Nxd7 18. Bxc4 Rc8 19. Na3 Nb6 20. b3 Qc7 21. Qd3 Nxc4
2. bxc4 Bxc4 23. Nxc4 Qxc4 24. Be3 Qxd3 25. Rxd3 Rd8 26. Ra5 Rab8 27. d5 Rd7
3. d6 Bf8 29. g3 Bxd6 *
   Here SF stucks at depth 22.
   Task manager shows that SF uses resources, and didn't freeze.
   In this game was used syzygy 3-4-5 pieces. But also without them latest version stucks at 22 ply.

Operating system

Windows

Stockfish version

Stockfish16 / fcbb02f

[Viren6]

I reproduced with the same dev version:

position fen 1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30
go depth 25
...
info depth 19 seldepth 40 multipv 1 score cp 158 nodes 602227 nps 1204454 hashfull 170 tbhits 0 time 500 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 d8d7 g1f2 h7h6 f2f3 f6f5 f3g3 d7d8 g3f2 d8d7 f2f3 d7d8 h3h4 h6h5
...
info depth 20 seldepth 44 multipv 1 score cp 168 nodes 210859338 nps 4161834 hashfull 250 tbhits 0 time 50665 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 h7h5 g1f2 h5g4 h3g4 g6g5 f4f5 d8d7 c5a3 d7d8 d1d2 d8d7 f2e1 d7d8 e1f1 d8d7
info depth 22 currmove a5a1 currmovenumber 1

Using dev version 1dfbde2 also gets stuck after:

info depth 21 seldepth 44 multipv 1 score cp 160 nodes 989167 nps 1469787 hashfull 207 tbhits 0 time 673 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1f2 d7d8 c5a3 d8d7 f2f3 d7d8 d1d2 e6f5 g4f5 h7h5 a3c5 d8d7 f3g3 d7d8 d2d1 d8d7 c5a3 d7d8 g3f2 d8d7 d1d2 d7d8
...

SF16 doesn't get stuck:

...
info depth 25 seldepth 46 multipv 1 score cp 126 nodes 1223530 nps 1718441 hashfull 280 tbhits 0 time 712 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 h7h5 g1g2 f7f6 g3g4 h5g4 h3g4 e6e5 f4f5 g6f5 g4f5 d8d7 g2f3 d7d8 f3e3 d8d7 d1d2 d7d8 d2d3 d8d7 e3e2 d7d8 d3d5 d8d7 e2d3 d7d8 d3e4 d8d7 d5d2 d7d8 e4e3 d8d7 e3f3
bestmove a5a1 ponder g8f8

[PavelChess]

I have prolonged the game by few moves:

1. Nf3 g6 2. d4 Bg7 3. e3 d6 4. c4 Nf6 5. Nc3 O-O 6. Be2 b6 7. Qc2 Bb7 8. O-O Nbd7 9. Rd1 e6 10. h3 d5 11. a4 c5 12. a5 cxd4 13. exd4 dxc4 14. a6 Bd5 15. Ne5 b5 16. Nxb5 Qb8 17. Nxd7 Nxd7 18. Bxc4 Rc8 19. Na3 Nb6 20. b3 Qc7 21. Qd3 Nxc4 22. bxc4 Bxc4 23. Nxc4 Qxc4 24. Be3 Qxd3 25. Rxd3 Rd8 26. Ra5 Rab8 27. d5 Rd7 28. d6 Bf8 29. g3 Bxd6 30. Ra1 Kf8 31. Rad1 Ke7 32. Bc5 Rbd8 33. Rxd6 Rxd6 34. f4 f6 35. g4 g5 36. f5 R8d7 37. Kf2 exf5 38. gxf5 h5 *

position fen "8/p2rk3/P2r1p2/2B2Ppp/8/7P/5K2/3R4 w - h6 0 39"
go depth 30

Here latest SF stucks at 17 ply.

The reason can be that SF tries to find a win, but, the the opponent has already built a fortress, and the computer may crash because of hardware failure before SF will see draw by 50 moves rule.

[Viren6]

Using dev version 99cdb92 also gets stuck for a while but does complete:

position fen 1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30
go depth 25
...
info depth 21 seldepth 44 multipv 1 score cp 134 nodes 731160 nps 1289523 hashfull 198 tbhits 0 time 567 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1f2 d7d8 f2f3 d8d7 f3g3 d7d8 h3h4 h7h6 h4g5 h6g5 g3f2 d8d7 f2e3 d7d8 e3e4 d8d7 d1d4 d7d8 d4d2
....
info depth 22 seldepth 47 multipv 1 score cp 157 nodes 98629776 nps 3199979 hashfull 379 tbhits 0 time 30822 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1f2 d7d8 f2g3 d8d7 d1d3 e6f5 g4f5 h7h5 g3f3 d7d8 f3e4 d8d7 e4e3 d7d8 d3d1 d8d7 e3f2 d7d8 f2f1 d8d7 f1g2 d7d8 g2h1 d8d7 c5b4 d7d8 h1g1
...
info depth 24 seldepth 46 multipv 1 score cp 157 nodes 100037351 nps 3185294 hashfull 461 tbhits 0 time 31406 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1f2 d7d8 f2g3 e6f5 g4f5 h7h5 c5a3 d8d7 d1d3 d7d8 g3f2 d8d7 f2f3 d7d8 f3e4 d8d7 d3d2 d7d8 a3c5 d8d7 e4f3 d7d8 f3e3 d8d7 e3e4 d7d8
info depth 25 currmove a5a1 currmovenumber 1

[peregrineshahin]

SF16 doesn't get stuck:

Stockfish 16 is also stuck after depth 25
changing the reproduction to go depth 26

[Sopel97]

I let the original position (1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30) run for a bit and it's still stuck on depth 22 currmove 1 after ~20 hours

[peregrineshahin]

Some data about the search before it gets stuck and after it gets stuck:

The completedDepth that gets stuck
`this->completedDepth == 21`

Hit #14: Total 898136113 Hits (Qsearch calls) - 33.4% of total hits
Hit #0: Total 1789284386 Hits (Search calls) - 66.58% of total hits:

Hits relative to search calls:
Hit #1: Total 232465473 Hits (Search ttCutoffs) - 12.99 %
Hit #2: Total 65359423 Hits (inCheck position) - 3.6%
Hit #3: Total 132938 Hits (excludedMove) 0.0074 %
Hit #4: Total 693280410 Hits (ttHits) 38.7%
Hit #5: Total 170633 Hits (new position eval in search) 0.0095%
Hit #6: Total 14764093 Hits (Razoring)  0.82%
Hit #7: Total 3626473 Hits (Futility pruning) 0.2%
Hit #8: Total 2448463 Hits (Null move search) 0.13%
Hit #9: Total 1382241 Hits (ProbCut) 0.077%
Hit #10: Total 677288142 Hits (Late moves reduction) 37%
Hit #11: Total 7055 Hits (FDS when LMR fails high) 0.00039 %
Hit #12: Total 1108091820 Hits (Full-depth search when LMR is skipped) 61.9%
Hit #13: Total 116 Hits (PV search) 0.00000648303874 %

Data for all completed depths before 21
`this->completedDepth < 21`

Hit #14: Total 62663619 Hits  (Qsearch calls) - 25.7 % of total hits
Hit #0: Total 180392949 Hits (Search calls) - 74.2% of total hits:

Hit #1: Total 40621273 Hits (Search ttCutoffs) - 22.5 %
Hit #2: Total 13625242 Hits 100 (inCheck position) - 7.55%
Hit #3: Total 68234 Hits (excludedMove) 0.037 %
Hit #4: Total 57946580 Hits (ttHits) 32%
Hit #5: Total 166466 Hits (new position eval in search) 0.092%
Hit #6: Total 2270827 Hits (Razoring)  1.25%
Hit #7: Total 3107863 Hits (Futility pruning) 1.72%
Hit #8: Total 1511612 Hits (Null move search) 0.13%
Hit #9: Total 199128 Hits (ProbCut) 0.83%
Hit #10: Total 102824080 Hits (Late moves reduction) 57%
Hit #11: Total 10694 Hits (FDS when LMR fails high) 0.0059 %
Hit #12: Total 75800618 Hits (Full-depth search when LMR is skipped) 42%
Hit #13: Total 2461 Hits (PV search) 0.00136 %

[Sopel97]

I let the original position (1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30) run for a bit and it's still stuck on depth 22 currmove 1 after ~20 hours

killed it after 30 hours total. Now running with 131072MiB of hash. Thought it would help (in case there's a lot of duplicated deep lines), but instead got stuck on depth 20.

[joergoster]

I doubt increasing the hash size will help.
Here a 2 min search with current master:

info depth 20 seldepth 40 multipv 1 score cp 154 nodes 683077 nps 493196 hashfull 5 tbhits 0 time 1385 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1g2 d7d8 g2g3 d8d7 h3h4 h7h6 h4g5 h6g5 g3f3 d7d8 f3f2 e6f5 g4f5
info depth 21 seldepth 40 multipv 1 score cp 154 nodes 200341237 nps 1669496 hashfull 11 tbhits 0 time 120001 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 f7f6 g3g4 g6g5 f4f5 d8d7 g1g2 d7d8 g2g3 d8d7 h3h4 h7h6 h4g5 h6g5 g3f3 d7d8 f3f2 e6f5 g4f5
bestmove a5a1 ponder g8f8

Search is stuck after 1 second, and even after 2 minutes the hashfull number is almost the same. This hints to that we are repeatedly starting a search, but almost never finish it to write the result into the Transposition Table.
Note, that we never return to the root and thus the previous PV gets displayed once again.

[PavelChess]

position fen "1r4k1/p2r1p1p/P2bp1p1/R7/8/3RB1PP/5P2/6K1 w - - 0 30"
setoption name threads value 24
setoption name hash value 2048
go depth 40

Stockfish dev-20231021-8366ec48 by the Stockfish developers (see AUTHORS file)
,,,
info depth 40 seldepth 71 multipv 1 score cp 118 nodes 117905211 nps 33939323 hashfull 42 tbhits 0 time 3474 pv a5a1 g8f8 a1d1 f8e7 e3c5
...

Stockfish dev-20240101-0fca5605 by the Stockfish developers (see AUTHORS file)
...
stop
info depth 21 seldepth 41 multipv 1 score cp 146 nodes 8593014540 nps 74362983 hashfull 5 tbhits 0 time 115555 pv a5a1 g8f8 a1d1
...

First version with dual NNUE:
Stockfish dev-20240107-a5a76a63 by the Stockfish developers (see AUTHORS file)
...
stop
info depth 26 seldepth 56 multipv 1 score cp 144 nodes 14647282219 nps 75001957 hashfull 10 tbhits 0 time 195292 pv a5a1 g8f8 a1d1
...

And yesterday very surprisingly a test game came by different way to very similar position with reversed colours, and black stuck again.

1. d4 Nf6 2. c4 e6 3. Nc3 Bb4 4. e3 O-O 5. Bd2 d5 6. Rc1 c5 7. a3 Bxc3 8. Bxc3 cxd4 9. Bxd4 Nc6 10. Nf3 Nxd4 11. Qxd4 Bd7 12. cxd5 Nxd5 13. Be2 Qa5+ 14. Nd2 Qa4 15. Qxa4 Bxa4 16. Bd3 Rac8 17. Ke2 Rfd8 18. Rhg1 Nb6 19. e4 Bc6 20. b3 Kf8 21. h4 g6 22. g4 f5 23. Ke3 Rc7 24. Nc4 f4+ 25. Ke2 Nxc4 26. Rxc4 Rcd7 27. Rc3 Rxd3 28. Rxd3

Latest abrok version 2048MB/24threads stucks at 24 ply:
stop
info depth 24 seldepth 39 multipv 1 score cp 236 nodes 12405234434 nps 81215854 hashfull 7 tbhits 0 time 152744 pv c6b5
...

[PavelChess]

Latest version without the issue:
Stockfish dev-20231027-b0658f0 by the Stockfish developers (see AUTHORS file)
Date: Fri Oct 27 17:24:25 2023 +0200
Timestamp: 1698420265
Introduce pawn structure based history
info depth 45 seldepth 77 multipv 1 score cp 110 nodes 280271611 nps 3371647 hashfull 35 tbhits 0 time 83126 pv a5a1

The issue began from next dev. version:
Stockfish dev-20231027-d30af4f by the Stockfish developers (see AUTHORS file)
Date: Fri Oct 27 17:32:19 2023 +0200
Timestamp: 1698420739
Rewarding Quiet Moves that Enable Razoring
stop
info depth 25 seldepth 34 multipv 1 score cp 142 nodes 6338685569 nps 4874760 hashfull 7 tbhits 0 time 1300307 pv a5a1

[peregrineshahin]

Well, since Stockfish 16 also gets stuck which is a commit before the version you are mentioning then the issue was still there but temporarily hidden.

Latest version without the issue: Stockfish dev-20231027-b0658f0 by the Stockfish developers (see AUTHORS file) Date: Fri Oct 27 17:24:25 2023 +0200 Timestamp: 1698420265 Introduce pawn structure based history info depth 45 seldepth 77 multipv 1 score cp 110 nodes 280271611 nps 3371647 hashfull 35 tbhits 0 time 83126 pv a5a1

The issue began from next dev. version: Stockfish dev-20231027-d30af4f by the Stockfish developers (see AUTHORS file) Date: Fri Oct 27 17:32:19 2023 +0200 Timestamp: 1698420739 Rewarding Quiet Moves that Enable Razoring stop info depth 25 seldepth 34 multipv 1 score cp 142 nodes 6338685569 nps 4874760 hashfull 7 tbhits 0 time 1300307 pv a5a1

[peregrineshahin]

luckily the issue still occurs with latest master after the latest merge wave. so we can test a rebased fix

[PavelChess]

In the last try for fix SF looks for fortress for both sides. I think that more efficient is to look for fortress only for opponent that has worse position, and in case of eval 0.00 to choose somehow one side for looking, for example by material balance.
Or maybe the program does it, and I didn't see because I am not really strong in programming ?

[Sopel97]

the search finished eventuall, this is with 128GiB of hash

info depth 24 seldepth 59 multipv 1 score cp 159 nodes 1256296412818 nps 3750194 hashfull 0 tbhits 0 time 334995009 pv a5a1 g8f8 a1d1 f8e7 e3c5 b8d8 d3d6 d7d6 f2f4 h7h5 g1f2 f7f6 g3g4 h5g4 h3g4 e6e5 f4f5 g6f5 g4f5 d8d7 f2f3 d7d8 d1d3 d8d7 f3g2 d7d8 g2f1 d8d7 f1e1 d7d8 e1d2 d8d7 c5b4 d7d8 d2d1 d8d7 d3d5 d7d8 b4c5 d8d7 d1e1 d7d8 e1f1 d8d7 c5a3 d7d8 d5d3 d8d7 f1f2 d7d8 f2f3 d8d7 a3c5 d7d8 d3d1 d8d7 f3g4 
bestmove a5a1 ponder g8f8

hashfull is at 0 after 1.2T nodes, nps is roughly 5 times higher than it should be. This indicates to me very few repeating positions that hit hash hard.

[PavelChess]

Latest version stucks at 28 ply.
I wish that the fix will be applied before Stockfish 17 release.

[peregrineshahin]

Tbh I ditched fixing this.. my proposed fix is very complexifying and it looks like this is caused because of the eval function not evaluating the position as a draw until it reaches to the 50 move rule which requires insane number of nodes to be searched with each depth.. best bet is to wish for a new net to evaluate this correctly.

[PavelChess]

I run matches using Cutechess 1.3.1. Stockfish often stucks before or at depth 25 at KBPK ending when it has rook pawn and bishop of wrong color. Even Syzygy bases 3-4-5 don't help.

Because of this I need to adjudicate draw manually, and also matches take more time.

Maybe this is less complicate to fix.

[Disservin]

@PavelChess if you have more positions feel free to post them here.

[vondele]

I think that's an example of what Pavel mentioned KBPK : 1k6/8/P7/8/K7/4B3/8/8 w - - 0 1

'stuck' on depth 20 with 4 pieces on the board (current master, all defaults).

EDIT: finishes the depth after a long while:

info depth 17 seldepth 31 multipv 1 score cp 31 nodes 2247956 nps 3469067 hashfull 57 tbhits 0 time 648 pv a4a5 b8a8 a5b6 a8b8 e3f2 b8a8 f2d4 a8b8 b6c6 b8a8 d4c5 a8b8 c6d5 b8a8 c5f2 a8b8 f2b6 b8a8 d5e4 a8b8 e4d3 b8a8 b6e3 a8b8
info depth 18 seldepth 30 multipv 1 score cp 28 nodes 3197429 nps 3525280 hashfull 77 tbhits 0 time 907 pv a4a5 b8a8 a5b6 a8b8 e3f2 b8a8 f2d4 a8b8 b6c6 b8a8 d4c5 a8b8 c6d5 b8a8 c5f2 a8b8 f2b6 b8a8 d5e4 a8b8 e4d3 b8a8 b6e3 a8b8 d3c3 b8a8 c3c4 a8b8
info depth 19 seldepth 34 multipv 1 score cp 39 nodes 7167553 nps 3923126 hashfull 85 tbhits 0 time 1827 pv a4a3 b8a8 a3b4 a8b8 b4c4 b8a8 e3d4 a8b8 d4b6 b8a8 a6a7 a8b7 c4b4 b7a8 b4b3 a8b7 b6d4 b7a8 d4c5 a8b7
info depth 20 seldepth 31 multipv 1 score cp 34 upperbound nodes 12312561 nps 4068923 hashfull 103 tbhits 0 time 3026 pv a4a3 b8a8
info depth 20 seldepth 36 multipv 1 score cp 31 upperbound nodes 2646947874 nps 4261092 hashfull 184 tbhits 0 time 621190 pv a4a3 b8a8
info depth 20 seldepth 36 multipv 1 score cp 28 upperbound nodes 4503629182 nps 4300610 hashfull 245 tbhits 0 time 1047207 pv a4a3 b8a8
info depth 20 seldepth 37 multipv 1 score cp 28 nodes 11730299633 nps 4255443 hashfull 245 tbhits 0 time 2756540 pv a4a3 b8a8 a3b4 a8b8 a6a7 b8b7 b4c3 b7a8 c3b3 a8b7 e3f2 b7a8 b3a4 a8b7 f2c5 b7a8 c5d4 a8b7 a4a5 b7a8 a5b5 a8b7 d4b6 b7a8 b5c4 a8b7 b6c5 b7a8 c4d5 a8b7 c5d4 b7a8 d5e5
info depth 21 seldepth 35 multipv 1 score cp 28 nodes 11730305801 nps 4255442 hashfull 245 tbhits 0 time 2756542 pv a4a3 b8a8 a3b4 a8b8 a6a7 b8b7 b4c3 b7a8 c3b3 a8b7 e3f2 b7a8 b3a4 a8b7 f2c5 b7a8 c5d4 a8b7 a4a5 b7a8 a5b5 a8b7 d4b6 b7a8 b5c4 a8b7 b6c5 b7a8 c4d5 a8b7 c5d4 b7a8 d5e5

[peregrineshahin]

I think that's an example of what Pavel mentioned KBPK : 1k6/8/P7/8/K7/4B3/8/8 w - - 0 1

'stuck' on depth 20 with 4 pieces on the board (current master, all defaults).

Hmm, my approach would solve this too, as the opponent can set on the position keeping playing the same move over and over again... needs rethinking..

[PavelChess]

Tbh I ditched fixing this.. my proposed fix is very complexifying and it looks like this is caused because of the eval function not evaluating the position as a draw until it reaches to the 50 move rule which requires insane number of nodes to be searched with each depth.. best bet is to wish for a new net to evaluate this correctly.

After upgrade of NNUE architecture the issue still exists. So the issue surely is caused by search algorithm.