Depth 245 anomaly during TCEC

[mstembera]

I was looking at this TCEC game
https://tcec-chess.com/#div=p&game=72&season=18&x=archive
and noticed that on move 67 SF reached our max depth of 245. This isn't so strange on its own except that the depth never went above 68 on any other move. The time used, nodes searched, and TB hits, all look normal. Does anyone have a plausible explanation or could this be an issue?

[joergoster]

There was also a big difference in depth between threads reported here #2620
Probably the thread with the highest depth was chosen by the thread voting. TB scores are getting stored with max depth, and one thread probably always took the TT cutoff and made it to MAX_PLY ...

[vondele]

TB could indeed be a reason for this. I couldn't yet reproduce it, but so far tried without TB.

[mstembera]

Maybe that's correct I'm not sure. The fact that it only happened in the position on move 67 and not any of the other similar positions just before or after seems odd regardless.

[mstembera]

Ok here is a game where it happens several times
https://tcec-chess.com/#div=p&game=16&season=18&x=archive

[vondele]

could we get the cutechess log for that game? Might be useful to figure out what's going on.

[vondele]

s18p_16live.log.txt

[Alayan-stk-2]

This happened again yesterday in another game after Stockfish had found a TB win.

[joergoster]

Are we sure the thread voting patch is working correctly for cases with very high depth differences?
https://github.com/official-stockfish/Stockfish/blob/master/src/thread.cpp#L240-L241

[vondele]

@joergoster not necessarily, or said differently, that thread will have a very high weight. I once tried to limit the depth difference, but those patches did not pass:
https://tests.stockfishchess.org/tests/view/5e9c2bd927460669d4042db0
https://tests.stockfishchess.org/tests/view/5e9c2bcd27460669d4042dae
https://tests.stockfishchess.org/tests/view/5e9c2bc127460669d4042dac

Some analysis why this is happening is still missing.

[joergoster]

@vondele But that seems more like a bugfix, no?

But first, however, it must be fully understood what is going on.

Edit: In TB win positions we shouldn't accept threads with lower scores, however!

[vondele]

Yes, we need to fully understand before we can say something is a bug fix, or test it as a bug fix.... so testing as an Elo gainer was a quick workaround ;-)

I agree that in TB win positions we should use the score, rather than a vote (but I think we do that already).

[vondele]

This seemingly still happens in quite recent times
https://www.chess.com/computer-chess-championship#event=ccc-16-blitz-finals&game=1
https://www.chess.com/computer-chess-championship#event=ccc-16-blitz-finals&game=22
https://www.chess.com/computer-chess-championship#event=ccc-16-blitz-finals&game=23
https://www.chess.com/computer-chess-championship#event=ccc-16-blitz-finals&game=54
...
shows this again.

[dav1312]

Happened again
https://tcec-chess.com/#div=sf&game=84&season=22

[ghost]

depth 245 is a maximum depth of stockfish
it was happened when adjustedDepth has bug or problem in main source code

Depth adjustedDepth = std::max(1, rootDepth - failedHighCnt - searchAgainCounter);
bestValue = Stockfish::search<Root>(rootPos, ss, alpha, beta, adjustedDepth, false);

see search.cpp , line 372 & 373 from this link :

https://github.com/official-stockfish/Stockfish/blob/master/src/search.cpp#L372
https://github.com/official-stockfish/Stockfish/blob/master/src/search.cpp#L373

[vondele]

so, some more investigation of the games https://tests.stockfishchess.org/tests/view/6272a6afc8f14123163c1997 some of these high depth happen also there, so it doesn't require TB. It doesn't happen in a single threaded run (https://tests.stockfishchess.org/tests/view/626abd7e8707aa698c0093a8), so it seems purely related to threading.
(note that these are depth 245 in the middle of a game, they can happen without concern if there is a mate announcement, or stalemate, 3fold on then next move).

So somehow an interaction between threads, but at least we can take TB out of the picture.

[miguel-l]

I'm not too familiar with threading code, but does the following make sense:
Is it possible that this has something to do with Threads.increaseDepth ?

As far as I can tell, this is handled only by mainThread and updated only at the end of its current iteration. My thought is if it gets stuck in resolving a fail or stuck in some search explosion while increaseDepth is false, then some other thread could be looping over the same rootDepth until they reach 245.

[ruicoelhopedro]

Two examples where this issue happens: depth245.zip
Logs with both regular debug log file and with per-thread search logs, reproduced locally.

For one of the cases, quick plot of the completed depth over time for each thread:

[mstembera]

I'm not sure if this is related but in the current TCEC swiss SF announced mate in 73 even though it only reported depth 39 and select depth 61. This was on move 53 of this game https://tcec-chess.com/#div=sw3&game=93&season=22

[vondele]

looks like @miguel-l has identified the cause in the test https://tests.stockfishchess.org/tests/view/62aee644e7ee5525ef88c15a

In discord:
Disservin
—
Today at 9:47 AM
I just checked the first 756 pgns and the patch has no weird 245 depths other than near checkmate or draw, base has 422
422 (some of them are also checkmate and draw depths) but shouldnt be that many < 10

I can see how these changes could be related.

[vondele]

@ruicoelhopedro any chance you could repeat your analysis, printing out an additional number of each thread, namely the adjustedDepth in addition to the rootDepth?

[vondele]

So, explanation being discussed at https://discord.com/channels/435943710472011776/813919248455827515/991975708953935903
basically, while the master thread has a hard time to resolve a given depth with increaseDepth false, other threads basically search at constant depth, which gets faster and faster over time. As a result their 'apparent depth' goes to the max 245 quickly.
I still have to look carefully at how this is actually fixed.
It might not be too bad what's going on right now, but it will screw a bit the thread voting scheme.

It's also consistent with the earlier posted graph where mainthread is stuck at a given depth.

and of course that's @miguel-l explanation.

[ruicoelhopedro]

@ruicoelhopedro any chance you could repeat your analysis, printing out an additional number of each thread, namely the adjustedDepth in addition to the rootDepth?

I've started some games with the extra field. I'll post the logs when the issue is reproduced.

[ruicoelhopedro]

Here they go, got a bit lucky as this happened in 3 games: depth245_adjustedDepth.zip

[vondele]

so, the logs show that what we think is happing is actually happening. The increaseDepth is false, adjustedDepth is constant, and the rootDepth just reaches the maximum.

[miguel-l]

Just want to have these somewhere I can find it (and also in case others might want to use or improve upon).

For downloading sample pgns from a test
for i in {0..99}; do wget "https://tests.stockfishchess.org/api/pgn/62aee644e7ee5525ef88c15a-${i}.pgn"; done

For manual checking
cat *.pgn | grep "White\|Black\|\/245" | grep -B 2 "\/245" | less