Random PR test comparison failures in DD4Hep workflows

[makortel]

We are seeing infrequent comparison failures in DD4Hep workflows in tests of unrelated PRs. The purpose of this issue is to track their frequency and progress to find out the cause.

[makortel]

assign geometry

[cmsbuild]

New categories assigned: geometry

@Dr15Jones,@cvuosalo,@civanch,@ianna,@mdhildreth,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @dpiparo, @silviodonato, @smuzaffar, @makortel, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

One occurrence is here #33526 (comment)

[makortel]

Another one #33534 (comment)

[makortel]

Here is another #33549 (comment)

[slava77]

is there a way to track which machine was used in each case?
This is in case there is some architecture dependence (me being reminded of existing differences in reco/miniAOD for e.g. tensorflow's mkl dependency)

[VinInn]

Did anybody run valgrind on those DD4HEP wf?

[mrodozov]

I asked last week on two PRs for externals the same thing
cms-sw/cmsdist#6822 (comment)
cms-sw/cmsdist#6787 (comment)

[makortel]

Here is another #33550 (comment)

[cvuosalo]

This workflow was stable for the last few weeks, but now it is unstable again. I think some recent change either re-activated an old bug or introduced a new stability problem. I will submit a PR to remove it from the PR tests for now.

[cvuosalo]

Looking through the recent history of merged PRs, I think the recent ROOT update might be a suspect for triggering the instability:

cms-sw/cmsdist#6835 ROOT change (April 23)
cms-sw/cmsdist#6826 DD4hep change (April 21)

That said, the instability seems to be around the 10% level. Many PR tests pass successfully with this workflow.

[cvuosalo]

@VinInn > Did anybody run valgrind on those DD4HEP wf?

Yes, I ran it back on March 5: #32963 (comment)
There have been changes to DD4hep and ROOT since then.

[cvuosalo]

PR #33568 submitted to remove the DD4hep workflow from the PR tests.

[cvuosalo]

+1

[cmsbuild]

This issue is fully signed and ready to be closed.

[cvuosalo]

The 11634.911 DD4hep workflow is now stable. See #34003. This issue can be closed.