recoConversion comparison instability in WF 23234.0

[kpedro88]

Spurious differences in recoConversion quantities have started appearing in WF 23234.0 (2026 D49):

Observed in the following PR tests so far:
#32324
#32326
#32327

[kpedro88]

assign reconstruction,upgrade

[cmsbuild]

New categories assigned: upgrade,reconstruction

@slava77,@perrotta,@jpata,@kpedro88 you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @kpedro88 Kevin Pedro.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[guitargeek]

The PR tests you quote use CMSSW_11_3_X_2020-11-29-0000, right?

Would that mean that the effects of all PRs merged since then would show up in the comparisons

I'm just worried that this issue might be related to my PR #32266 that indeed touched the conversion producer code, and this PR was one of two that was merged since the release that was used for the comparison baseline I think:

https://cmssdt.cern.ch/SDT/html/showIB.html

However, #32266 only moved code from some .h to .cc files and occasionally did the getByToken() to getHandle() modernization. So I don't know if it's actually related.

[kpedro88]

Usually, the bot posts a notification if it merges additional PRs on top of the IB. I don't see that in this case.

[perrotta]

The PR tests you quote use CMSSW_11_3_X_2020-11-29-0000, right?

Would that mean that the effects of all PRs merged since then would show up in the comparisons

No, only the PRs merged before CMSSW_11_3_X_2020-11-29-0000 can be responsible. Your own was only merged later on in CMSSW_11_3_X_2020-11-29-2300

[slava77]

is there any indication of the problem before CMSSW_11_3_X_2020-11-29-0000?
I'm thinking that the point was the switch to 11_3_X (new gcc version and what not) and that everything after we switched until 11-29-0000 was just not visible as a problem due to the issues with IBs and the tests.

[kpedro88]

I hadn't noticed the problem before today (but was offline for the holiday the past few days). The gcc version change is worth checking.

[smuzaffar]

@kpedro88 , is this still an open issue?

[kpedro88]

@smuzaffar can you fix the bot so it stops removing my labels?

I don't recall observing these spurious differences recently, but I'm also not aware of a particular PR that would have fixed them. Current L2s are probably more attentive to this than I am.

[jpata]

This happened again in #36910 for 12_1_X, ZMM13TeV2017wf10042p0.