Don't use the contempt setting in the analysis mode

[atumanian]

The problem with using the current implementation of contempt for analysis is that the setting is applied from the point of view of the side to move. While in the analysis mode the side to move often switches. This leads to inconsistent analysis and incorrect functioning of the TT.

We should probably add a boolean option which makes possible to apply the contempt setting always from the white's perspective (Komodo and Houdini have White Contempt and Analysis Contempt respectively for this purpose). It probably needs discussion. But in any case disabling the contempt for analysis is a necessary step.

Bench: 5098576

[atumanian]

@mcostalba said:

@atumanian the correct line is that user should set contempt zero for analysis. Please note that this is true for any engine at any time, and (to me) it is acceptable to think this fact is in the knwoledge of users that usually do chess analysis.

The new fact here is that before contempt was at zero by default now it should be explicitly set.

I disagree.

I've pointed out serious problems with binding the contempt to the side to move in analysis. Users aren't obliged to know all the engine options. It's not acceptable that default settings produce buggy behaviour. Remember also that Stockfish is used for live analysis by such services as Chessbomb. Should we tell all of them to change the setting?
It's not true for neither Komodo nor Houdini - they both don't use contempt for analysis by default.

[jhellis3]

This is actually a very bad thing. It takes control away from the end user, and makes it so what they see is not what they get. What if the end user wants to use contempt in analysis? They would have no way to do this. Furthermore, the setting would then become disingenuous with undocumented (to the end user) behavior. Having the current contempt displayed correctly is reasonably sufficient. If someone is so deficient of intellect that they are unable to learn about this, then it is unreasonable to expect them to be able to get anything out of analysis in the first place.

[Stefano80]

I strongly assume that already now professional correspondence players use contempt. What should they do with this patch?

[Nordlandia]

Contempt should automatically be set to zero for analysis.

[atumanian]

@jhellis3, @Stefano80,
I agree that user should be able to set contempt for analysis, but not for the side to move, but always for white. I've written about it in the description of the PR. This patch fixes the bug introduced with the change of the default contempt. A contempt setting for analysis can be introduced with the next PR.

[jhellis3]

There is no bug in the current implementation AFAICS. It does what it says it does. If you want to extend that functionality to include the ability to selectively enable/disable contempt for White or Black, that is fine. But taking control away from the end user and forcing them into doing what you like is very wrong. What if I want contempt enabled for both sides? You don't get to decide what I want. I do.

[Vizvezdenec]

Idk I think that people that use contempt for analysis are smart enough to set it to 0... If they want. Disabling ability to change it is just bad.

[atumanian]

@jhellis3, I've described the problem of using the current implementation of contempt for analysis in the first paragraph of the PR message. You can also see @syzygy1 's posts here, where he explained the problem in detail.

The point is this. If you start analyising the starting position (say), then a +7 contempt bonus is given to white. If you now move e2e4 and continue analysing, suddenly a +7 contempt bonus is given to BLACK. This makes no sense at all. It also screws up the TT.

[atumanian]

The problem with using contempt for analysis is solved with this patch, which introduces an Analysis Contempt option like in Houdini.

[snicolet]

As I understand it, people seem to be afraid of hash pollution when analyzing positions of consecutive plies for the same game, alternating searches from root-color-is-White and root-color-is-Black point of view. Is that correct?

I would like to get a better grasp of the issue, just to convince myself that there is a problem: could anybody show us a concrete sequence of UCI commands to type in a terminal to exhibit the effect, and explain us why the chosen moves are bad (losing moves in a draw position, or drawing moves in a won position, stuff like that)? Thanks in advance!

If there is really such a proven effect, then it would also be nice to be able to quantify it in some sense (what is the probability that the effect appears? How much Elo does it cost? Does the effect disappear if we search to a certain depth? or after a certain amount of time? ...)

On a subjective/personal side, and if we agree that there is a problem, my preference would be an automagic solution rather than relying on another UCI option which would come with another set (or stream :-) of angry comments. This does not mean that I exclude the "UCI option" option, but only as a last resort.

For instance, would it be an acceptable compromise for @syzygy1 if we used two different hash lines for the "root-color-is-White" and the "root-color-is-Black" analyses? This is a little bit similar to what we do when use different hash cells to store the same position with Black-to-move and White-to-move, after a null move or a triangulating transposition.

What about adding one statement like the following around line 199 of search.cpp?

    rootPos.set_key(us == WHITE ? std::min(rootPos.key(), rootPos.key() ^ contempt)
                                : std::max(rootPos.key(), rootPos.key() ^ contempt));

[jhellis3]

That is not a problem. You think it is a problem. When one is analyzing an unclear position, very often both sides have winning chances. It is not reasonable to expect one's opponent be willing to settle for a draw in these situations.

Basically, there are 3 possible scenarios:

1. One side is clearly winning (a small contempt value is meaningless here as it will not interfere with any "saving" move).

2. Only one side has winning chances but a draw is also possible (the only real scenario where this might be a problem).

3. Both sides have winning chances (you actually want contempt applied to both sides here despite the unfortunate interaction with the TT).

The patch you linked to is not a complete solution either.

[jhellis3]

Stéphane seems to be on the right track. Other possibilities are to simply clear the hash on position change in analysis when a non-zero contempt value is detected or maintain 2 separate TTs half the size. However, I think it needs to be demonstrated that this TT issue can actually result in some real world pathological behavior. SF already uses 16-bit keys.... and yet.

[ceebo]

I do think having a non-zero default contempt that switches sides in analysis mode is a bit weird (just my personal opinion) but, like @snicolet says, the thing I find most ugly about it is the pollution of the hash table.

How about something like this though?

  if (rootPos.side_to_move() == WHITE || Options["White Contempt"])
      Eval::Contempt = make_score(contempt, contempt / 2);
  else
  {
      Eval::Contempt = -make_score(contempt, contempt / 2);
      rootPos.apply_black_contempt_key();
  }

I think it allows everyone to find a setting they prefer and the hash pollution issue is prevented. The only thing I haven't checked is how this will work with the pos_is_ok() stuff in debug mode --- having the hash key of a position depending on the side to move in the root position is something new.

[syzygy1]

If professional players are using non-zero contempt with Stockfish today, then they just don't realise the effect of it and that it's counterproductive. And it would not surprise me if some really do use it, because it is not immediately obvious at all that there is a problem with it (which may be why there is still some sort of denial of the problem here).

First a reality check: both Komodo and Houdini have fixed the issue. There is a reason for that. They don't just "take away control from the user" for nothing. (In fact they don't take away control at all: they turn contempt into something that is useful even in analysis.)

To see what current non-zero contempt does when analysing the root position, see here: #1361 (comment)
Look at the scores reported when black is to move. They start with -156 (because the TT was filled with contempt for white), then switch to +80. (Of course setting contempt to 100 magnifies the effect.)

So clearly there is TT pollution. Basically, the engine is using two different evaluation functions depending on whether it's white's or black's move at the root.

I guess it would be possible to fix this in an ugly way by somehow splitting the TT in two. But is that really what we want?

What would a professional chess player use contempt for during analysis?

I don't think a professional chess player would typically analyse a position to see if white can win when white is to move at the root and to see if black can win when black is to move at the root.

I think a professional chess player expects Stockfish to use the same evaluation function independently of the side to move at the root. If he sets contempt, it is because he wants to see if white can win and, equivalently, if black can defend the draw. Or the other way around. But normally not both, and certainly not dependent on whether the position on the board happens to have white to move or black to move (which is totally arbitrary when playing through variations).

Setting contempt to zero during infinite analysis takes just one line of code or so. Compared to SF's current state, it takes away a useless degree of freedom (since contempt is now useless for analysis).

To win back the freedom, one could automatically force contempt to "white contempt" during infinite analysis, but this might be confusing. An "Analysis Contempt" option (as in Houdini) or a "White Contempt" option (as in Komodo) seems better to me. (If not set, contempt is forced to 0 during analysis; if set, contempt is forced to be from white's pov.)

To be even more complete, one could reintroduce UCI_Analysis. A UCI-compliant GUI detects that option, hides it from the user, and automatically sets it whenever the user is doing analysis. (One could think of annotating a game at 10 seconds per move, for example. So analysis but not infinite analysis.) I guess one could do without a UCI_Analysis option but it boggles my mind why - as always - two more lines of code are too much of a hurdle for something that is an objective improvement to user experience. (Btw, SF used to have UCI_Analysis - probably from Glaurung but I"m not sure - exactly because it had an asymmetric evaluation. The asymmetry was removed if UCI_Analysis was set by the GUI.)

[atumanian]

TT pollution isn't the only problem. Another problem is inconsistency in the analysis. We set a positive value of contempt to account for the difference in strength of the players. When we do analysis for the white side, the contempt setting means that white is stronger than black. But when we make a move and continue analysis, how can black suddenly become stronger?

[syzygy1]

@atumanian yes, that's why I say current contempt is useless for analysis.

Perhaps many here think analysis == running infinite mode on a single position for minutes or hours or days.

I think that, in the first place, analysis == the user exploring variations by going through (half-)moves back and forth.

[atumanian]

@syzygy1, completely agree with you.
By the way, I've started a topic on Fishtest to discuss ways to implement contempt in the analysis mode: Contempt setting for analysis

[jhellis3]

If professional players are using non-zero contempt with Stockfish today, then they just don't realize the effect of it and that it's counterproductive.

Bullshit. And it isn't counterproductive. It depends on what the end user wants.

But when we make a move and continue analysis, how can black suddenly become stronger?

Well, for one you are using contempt, which overestimates your position for non-draws, so obviously the non-side to move is stronger than what you are seeing. And this will remain the case regardless of your "fix", the difference will just shrink by half.

But both of you are treating the end user as if they are complete morons who are permanently unable to grasp what is going on, which is a strawman.

How about someone post a position which actually causes a meaningful, pathological problem first...

[mcostalba]

@atumanian the value of this PR is mainly in starting a discussion. The patch itself is a design mistake because, as pointed out alredy by other people, we really don't want to be so smart to tweaking contemt value behind the user.

If the user explicitly sets contempt to a specifc value through a UCI optoin, it is very bad design to silently change it by mean of an unrelated path. So this patch is deeply wrong and can't be committed as is.

Nevertheless I will let this open for a while just to encourage discussion.

[AlexandreMasta]

As said by some and already understandable...you that want this lock on contempt are considering users as morons. The concerns are not as critical as you think. This patch is not necessary.

[syzygy1]

If it is a design mistake, then it was a design mistake in early SF and probably in Glaurung.

I guess it is also a design mistake in Komodo and Houdini and probably the majority of other engines.

I guess all those engines treat their users as morons.

[syzygy1]

I wonder if anyone against fixing the issue is able to explain in clear terms what the issue is.

[syzygy1]

@mcostalba No, the user does not explicitly set a contempt value now that the default is non-zero.

But even if he does set it explicitly, contempt is not doing what the user expects it to do, because it continuously changes its sign.

[syzygy1]

@mcostalba sorry, I misread what you wrote! I guess I missed the "if".

[lantonov]

Now the user has the option to change the contempt before each ply or make a ply with several contempts, if he so wishes

[atumanian]

@jhellis3 said:

How about someone post a position which actually causes a meaningful, pathological problem first...

@syzygy1 has already provided a position demonstrating the issue in this comment.

Well, for one you are using contempt, which overestimates your position for non-draws, so obviously the non-side to move is stronger than what you are seeing. And this will remain the case regardless of your "fix", the difference will just shrink by half.

I don't understand what "the non-side to move is stronger than what you are seeing" means. Do you agree that the contempt setting expresses the a priori difference of the players' strengths?

[atumanian]

@mcostalba, what do you think about fixing the bug in the same way as that of Komodo or Houdini?

[syzygy1]

@lantonov
Indeed, but that seems rather involved.

We could instead rely on the GUI to do it, but typically GUIs don't do this. As far as I'm aware. GUIs leave this to the engine, and that seems unlikely to change.

[atumanian]

@syzygy1, I agree it is a bug in Arena. And what do think about live analysis by Chessbomb? What do think about returning the default value of Contempt to 0 to avoid potential problems with programs which don't properly comply with UCI?
@Nordlandia, @anshulongithub, @MichaelB7, your opinion about the question (what condition to use to determine that we are in the analysis mode, and whether to return to 0 as the default of Contempt) would also be very desirable.

[syzygy1]

Chessbomb should set the contempt value to zero.
If they don't do that, then that suggests that @mcostalba might want to reconsider what he wrote earlier:
#1361 (comment)

This is a general usage rule for any engine and (to me) it is assumed to be in the knwoledge of the users that perform engine analysis. IOW if someone uses an engine to analyze position it is fair to assume he knows something about what he is doing, especially if he does routinely analisys eventually with different engines.

and

It is resonable to assume that an user that does analysis (so uses the engine in somewhat advanced way and can be assumed to be a power user), he knows what contempt is.

It is clear that many even advanced users do not realise what contempt does to analysis or are simply not aware of contempt at all.

If we care about users, the sensible thing to do, in my view, is to either

• set default contempt to zero and leave it to the advanced users to change the setting or
• force contempt to zero if analysis mode is detected (and optionally provide a setting to keep it enabled).

Btw, a related issue is: what to use on fishtest?

[vdbergh]

@syzygy1 At least the question what to do on fishtest can be given an objective answer. The issue is whether contempt (in both engines) increases/decreases BayesElo differences. This can be measured, using two versions of SF that are not too close in strength (if the strength difference is too small the effect of contempt will be too small to be reliably measurable).

[atumanian]

I now think setting the default of Contempt to not zero was a mistake. The only rational reason for this is to improve our position in rating lists such as CCRL. But it creates the problem of having biased analysis in chess interfaces which don't properly determine the analysis mode. Relying on users isn't reasonable because they aren't obliged to know all the non-standard settings.
Therefore I think we should now request to pull the version of Analysis Contempt with only Options["UCI_AnalyseMode"] for identification the analysis mode. Next, the problem with interfaces which don't use UCI_AnalyseMode properly can be solved in either of the two ways:

• set the default value of the Contempt settìng back to 0;

• add Limits.infinite to the condition identifying the analysis mode.

[snicolet]

@atumanian

I would like to get a better grasp of the issue, just to convince myself that there is a problem: could anybody show us a concrete sequence of UCI commands to type in a terminal to exhibit the effect, and explain us why the chosen moves are bad (losing moves in a draw position, or drawing moves in a won position, stuff like that)? Thanks in advance!

[atumanian]

@snicolet, see syzygy1's comment. The problem is that when we switch the side to move the evaluation function and the displayed scores become different.

[snicolet]

@atumanian
Yes. Why is it a problem ?

[vdbergh]

@snicolet It seems obvious to me that analysis scores which jump up and down randomly are not desirable. Unless one is not interested in analysis at all....

[AlexandreMasta]

@vdbergh just set contempt value to "0".

@atumanian works as intended. Engine uses contempt 20 as default.

@atumanian the default setting should be always the optimum value. In this case was already bissect and contempt 20 gives the best elo score from all settings without regression, so it must be the default.

[syzygy1]

What do users expect?

Houdini:

By default a non-zero contempt is only used during game play, not during infinite analysis. If you enable the Analysis Contempt checkbox, Houdini will also take into account the contempt for infinite analysis.

Komodo:

In analysis mode, Komodo internally sets Contempt to zero unless “White Contempt” is turned on.

It seems Rybka has separate "Contempt Play" and "Contempt Analysis" settings.

edit: Stockfish (until the removal of asymmetric king safety parameters) is in this list as well.

[vdbergh]

@AlexandreMasta You are completely wrong. The default setting should give the best user experience. In analysis the user wants sensible scores which in any case should not randomly jump up and down. A (perhaps) ultra tiny elo increase (likely due to some flaw in SF) is completely irrelevant in this context.

[snicolet]

@vdbergh

Even with contempt=0, there is variability...

May I show you something with current master, 1 thread, contempt=0 ? Type the following in a terminal (test is deterministic):

./stockfish
setoption name Contempt value 0
position startpos moves e2e4
go depth 20 => score cp -27 pv e7e5
go depth 20 => score cp -25 pv e7e5
go depth 20 => score cp -18 pv e7e5
go depth 20 => score cp -17 pv e7e5
go depth 20 => score cp -41 pv e7e5
go depth 20 => score cp -42 pv e7e5
go depth 20 => score cp -53 pv e7e5

So even on one thread with contempt=0, Stockfish evaluates randomly the Spanish from -0.17 to -0.53 pawns? What is the sensible score of the Spanish opening? What should we store in the hash table at depth 20?

There is no absolute value for a position in modern chess engines, they are just non-deterministic statistical filters on the game tree, proposing a set of moves in a position with a blurry note. With Contempt=20, the shape of the filter is a little bit different, but the output moves are just as good as for Contempt=0.

[syzygy1]

@snicolet
It is not about the output moves but about the scores returned.

We know that the search returns different scores if the TT is filled with better values. So repeating "go depth 20" should indeed lead to varying and supposedly more accurate scores. To get even more accurate scores, let the engine think longer.

Maybe you think scores returned by SF are worthless anyway, but I think many users disagree.

[syzygy1]

If we agree that Stockfish gets it right only "statistically", then we should also agree that statistics should start from unbiased results.

[NKONSTANTAKIS]

Master should be the default for analysis because it is superior.

The fluctuation of the reported eval is not random but affects the side to move. This helps the engine but might be a little uncomfortable to humans. The simplest solution would be for the human who wants to get an as much as possible objective impression to subtract a little, around 0.15, from the side to move. If this feels cumbersome, then it would be better to hard code SF to report -0.15 from the score it uses than weakening SF by using C0.

C0 had some kind of "bug" which was reporting too many 0.00 evaluations and professional players and corr players had hard time selecting which of those variations would be better for them. Now this is greatly fixed and the objectively best variation will have the best score. The score will be less objective but just by subtracting 0.1-0.2 from it, it will also be more objective than before.

Of course having the option of choosing which setting someone prefers on the fly is a good thing.
Why to force people to use something when they can have both with the click of a button?

[AlexandreMasta]

@syzygy1 Users expect to have the best move for a given position. If the best move is achieved by having a little of contempt so be it.

@vdbergh You are completely wrong. The default setting should give the best move in a given position.

I can say and tests should confirm that a little of contempt (I use 7) with the new improved contempt option is stronger than master C0 (I know it is weird but it is how it is).

If C7 or C20 is stronger than C0 it shall be the default. Very simple logic here.

[atumanian]

I've created a new pull request with the better solution to the problem using the new Analysis Contempt setting. Should we close this PR in favour of the new one?

[vdbergh]

@AlexandreMasta I think you do not really understand what you write.

Assume for the sake of argument that contempt does indeed yield 0.5 elo in self play (in view of the tests). This roughly corresponds to gaining 1s per 3min analysis. This is totally undetectable. Moreover, any possible elo gain due to C20 is likely destroyed (and worse) due to hash table pollution by going back and forth (40cp bias!).

So your implication that SF with C20 suddenly finds "better moves" in analysis is a totally unrealistic description of reality.

[atumanian]

@NKONSTANTAKIS, subtructing a constant value from all scores only hides the problem from the user, not solves it. For example, in a position with a forced draw the score is always 0.00, regardless of contempt.

C0 had some kind of "bug" which was reporting too many 0.00 evaluations and professional players and corr players had hard time selecting which of those variations would be better for them.

Why is this a bug?

[syzygy1]

@NKONSTANTAKIS
It is absolutely impossible that an analysis with contempt = 0 is "weaker" than an analyis with contempt = +/- 20. Think about it: would BOTH +20 AND -20 contempt (from white's point of view) be "stronger" than simply 0 contempt? That's ridiculous.

I haven't really tried it out yet, but I see no reason why any persistent "0.00" problem does not still exist. It has just been shifted to what used to be -0.15 and +0.15 positions (somewhat randomly depending on whose side to move it is at the root!).

[AlexandreMasta]

@syzygy1 No. That´s not ridiculous given the way the engine is searching moves right now. Somehow the engine is searching winning lines with a little of contempt that wasn´t before with contempt 0. I´m surprised too...but this is what it is. Stop pretending you are "Mr. Know It All". Your ego is so inflated that you want to go against the tests. If this is some kind of SF bug or not I let for further researchs to explain the fact.

[atumanian]

@AlexandreMasta, results from playing cannot be easily generalised to analysis. It has already been explained that during analysis the sides are often switched, unlike during playing.

[AlexandreMasta]

@atumanian already understood that...I DO ANALYSIS.

I just go to the UCI options and set "Contempt" 0. After that I start my analysis. I´m not having any problem at all.

[syzygy1]

@AlexandreMasta I am sorry for your compensatory needs, but to be frank with you, I cannot be bothered too much.

[AlexandreMasta]

@syzygy1 You don´t have nothing to be sorry...did I hurt you? I don´t need any. But you seems to think you have the truth in all of your posts, what is annoying. I´m just trying to soften your inflated ego behavior a little.

[NKONSTANTAKIS]

Drawish behavior is killing chess, chess is a game not a science. This new SF can make everyone play more interesting chess and enjoy it more and since SF is the best engine most follow it and the whole world will welcome this change. And its +10 elo to SF8 and +2.5 elo better. So its better everywhere. But even if it was worse, it would still be a good thing. Those who don't want it know how to not use it.

[syzygy1]

Sure @NKONSTANTAKIS , but analysis is a very different story. I'm not saying contempt can never be useful for analysis (but I tend to think the old draw-score contempt was a more useful instrument for the purpose of analysis as its results were much easier to interpret).

[NKONSTANTAKIS]

I think all options should be available for the user because all have merits.

[atumanian]

I close this PR in favour of #1376.