IB Failure with workflow 23234.103

[qliphy]

It appeared firstly from IB CMSSW_12_2 2021-10-31-2300, and persists up to most recent IB.

Previously we thought it is due to #35744 , and #35941 may fix this as a byproduct
#35744 (comment)
The PR test with #35941 enabled with multithreading worked well with 23234.103
#35941 (comment)

However, after merging #35941 the issue persists up to most recent IB. We may need to wait to see the next IB results.

Let's open a thread here to track this issue.

Adding @drankincms

p.s. I tried several local tests and they work well with
runTheMatrix.py -l 23234.103 --job-reports -t -4 --ibeos

[cmsbuild]

A new Issue was created by @qliphy Qiang Li.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[qliphy]

assign dqm,l1

[cmsbuild]

New categories assigned: dqm,l1

@jfernan2,@ahmad3213,@rvenditti,@emanueleusai,@rekovic,@pbo0,@cecilecaillol,@pmandrik you have been requested to review this Pull request/Issue and eventually sign? Thanks

[Dr15Jones]

I believe I've traced down the problem to a thread safety issue in ROOT.

The call to QuantilesX does

https://github.com/root-project/root/blob/de7bcdff077be4ec08892c1d8349d9f8e62dd420/hist/hist/src/TH2.cxx#L2460-L2463

and in DoQuantiles there is a call to DoProjection which uses the histogram name "tmp".
https://github.com/root-project/root/blob/de7bcdff077be4ec08892c1d8349d9f8e62dd420/hist/hist/src/TH2.cxx#L2531

It then checks gROOT if that histogram name already exists and then attempts to re-use it
https://github.com/root-project/root/blob/de7bcdff077be4ec08892c1d8349d9f8e62dd420/hist/hist/src/TH2.cxx#L2189-L2199

So if any other module is calling another ROOT function which also uses the "tmp" histogram then the histogram could be reset right as the call to ComputeIntegral happens.

The upshot is it is not safe to call QuantilesX or QuantilesY and you will have to find another way to fill the histograms.

FYI @pcanal.

[Dr15Jones]

NOTE: It appears ProjectionX and ProjectionY have a similar problem if one is not careful to pass in unique names to the functions.

https://github.com/root-project/root/blob/de7bcdff077be4ec08892c1d8349d9f8e62dd420/hist/hist/src/TH2.cxx#L2391-L2394

https://github.com/root-project/root/blob/de7bcdff077be4ec08892c1d8349d9f8e62dd420/hist/hist/src/TH2.cxx#L2431-L2434

[drankincms]

@Dr15Jones thanks a lot for tracking this down! Ok, I will go ahead and implement QuantilesX() by hand. @qliphy I will go ahead and submit a PR to fix this once it's ready.

[Dr15Jones]

@drankincms it looks like if you first set ROOT to a unique new directory it will place the "tmp" into that directory which should isolate you from any other threads.

[Dr15Jones]

So the ROOT code does search gDirectory for an object with the name "tmp". It looks like you can temporarily replace gDirectory via

https://github.com/root-project/root/blob/52afd08119469dae2ebecf14fbfe7c05e8a756b3/core/base/inc/TDirectory.h#L73-L84

[drankincms]

@Dr15Jones Thanks a bunch for the tip, your TContext trick seems to fix this.

[jfernan2]

+1
I believe this issue is being fixed by #36067

[perrotta]

@qliphy I think we can close this (please reopen if I'm wrong)