Spurious differences appearing for 20024.0 in Jenkins DQM results, starting around CMSSW_8_1_X_2016-09-26-1500

[cvuosalo]

Some recent 81X Jenkins results show spurious DQM differences for Phase 2 workflow 20024.0. They show up as 1014 DQM plots with very tiny differences.

Here's a list of PRs that have been checked, with baseline and whether the spurious differences appear:
#15907, CMSSW_8_1_X_2016-09-27-1100, Yes
#15992, CMSSW_8_1_X_2016-09-27-1100, No
#15989, CMSSW_8_1_X_2016-09-26-1500, Yes

[cmsbuild]

A new Issue was created by @cvuosalo Carl Vuosalo.

@davidlange6, @smuzaffar, @Dr15Jones can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[cvuosalo]

assign reconstruction

[cvuosalo]

Spurious differences for 20024.0 have been replicated with #15989 with 20 events against baseline CMSSW_8_1_0_pre12.

[cvuosalo]

More spurious differences for 20024.0 in #16010 with baseiine CMSSW_8_1_X_2016-09-28-1100.

[cvuosalo]

More spurious differences for 20024.0 in #15996 with baseiine CMSSW_8_1_X_2016-09-28-1100 and
for #16009 also.
And for #16013 with baseline CMSSW_8_1_X_2016-09-28-2300.
Also #16021 with CMSSW_8_1_X_2016-09-28-1100.
Also #16041 with CMSSW_8_1_X_2016-09-29-1100.

[slava77]

@cvuosalo
let's try to narrow this down.
In pre12: please check if the differences are still present by

• rerunning using step1.root from the baseline
• rerunning using step2.root from the baseline

Thanks

[cvuosalo]

@slava77: The differences disappear when I used step1.root from the baseline for #15989 and baseline CMSSW_8_1_0_pre12, so it might be a GEN-SIM problem.

[slava77]

@civanch @ianna @boudoul
are you aware of any issues in the phase2 "D1" geometry setup that could introduce reproducibility problems at SIM level?

[civanch]

The problem should be fixed in #16069

[cvuosalo]

@civanch: #16069 was merged starting with CMSSW_8_1_X_2016-10-03-2300. However, the bug is still active in CMSSW_8_1_X_2016-10-06-1100 (see Jenkins for #16054 as an example).

[civanch]

@cvuosalo , concerning #16054 : for me there are comparison problems only in the work flow 20024.0, while all others are on a standard level; even for this WF number of differences is much less compared to the case, when bug was introduced.

[cvuosalo]

@civanch: The DQM differences for #16054 number about 1000 plots, which is about the same as for the PRs initially mentioned in the description of this issue, so I don't see any improvement.

[smuzaffar]

@cvuosalo , is this still a valid issue? can we close it?

[cmsbuild]

New categories assigned: reconstruction

@slava77,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cvuosalo]

@smuzaffar @slava77 This issue is so old that I don't think it is relevant anymore. We can close it if Slava agrees.

[slava77]

it's OK to close.

[cvuosalo]

Issue no longer relevant.