Non-reproducibility in workflow 11634.0

[makortel]

PR tests in #44624 (comment) showed 6032 differences in the comparisons of 11634.0 The differences seem to be pretty much everywhere.

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@sextonkennedy, @Dr15Jones, @makortel, @smuzaffar, @antoniovilela, @rappoccio can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

Another one in #44575 (comment)

[makortel]

Both of the tests mentioned so far were run on top of CMSSW_14_1_X_2024-04-17-1100. No PRs were merged in that IB (except one data PR), but the earlier IB was broken (because of the data PR not being included there), and in that IB the following PRs were merged

• Fix for EndPaths in DependecyGraph #44742
• Introduce Unified Particle Transformer AK4 jet tagger #44641

[makortel]

Some more, all using CMSSW_14_1_X_2024-04-17-1100

• [14_1_X] [Phase-2] Changed Phase-2 JME and BTV HLT paths to use the L1 P2GT emulator #44751 (comment)
• Adding PNET scores to offline HLT DQM #44754 (comment)
• Adding PUPPI jets and METs to offline DQM #44336 (comment)

I searched for PRs that were tested with surrounding IBs (from 04-15-2300 to 04-18-1100) but didn't see any other PR test to be affected

[makortel]

I checked PRs that used IBs up to CMSSW_14_1_X_2024-04-20-1100 in testing and did not see any such large differences.