backport of CASTOR rechitcorrector and noise simulation fixes

[hvanhaev]

This is a back port of changes already included in the master, as discussed in PR: #22424

This contains a fix in the CASTOR noise simulation, and updates in the rechitcorrector module.
It would be good to have these too in CMSSW_76X for analyses with 2015 data.

[cmsbuild]

A new Pull Request was created by @hvanhaev (Hans Van Haevermaet) for CMSSW_7_6_X.

It involves the following packages:

RecoLocalCalo/Castor
SimCalorimetry/CastorSim
SimGeneral/MixingModule

@perrotta, @cmsbuild, @civanch, @mdhildreth, @slava77 can you please review it and eventually sign? Thanks.
@makortel this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

• Backported from CASTOR reconstruction and simulation updates #22424

[slava77]

I don't think we even have a 7_6_X IBs set up.
@smuzaffar @fabiocos
please let me know if we do.
Thank you.

[smuzaffar]

@slava77 , correct we do not have 76X IBs running.
@fabiocos if this is needed then should we enable 76X IBs for a week?

[slava77]

@smuzaffar @fabiocos
can we revive a 76X IB?
I suppose that only one will be needed to run the tests for this PR (even if the tests are started or remade a few days after that IB is made)

[perrotta]

backport of #22424

[perrotta]

@hvanhaev : in case you want to proceed with this proposed backport, the same adjustments that guarantee the preservation of the current behaviour as in #23894 should be also ported here

[hvanhaev]

Ok thanks, I can update this PR too yes, should be ok. Cheers, Hans On 28 Aug 2018, at 15:14, perrotta <notifications@github.com<mailto:notifications@github.com>> wrote: @hvanhaev<https://github.com/hvanhaev> : in case you want to proceed with this proposed backport, the same adjustments that guarantee the preservation of the current behaviour as in #23894<#23894> should be also ported here — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#23895 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEgkF9XyGTqqu_8YvBLUfwzkdoKeDQooks5uVUImgaJpZM4VX7Dv>.

[cmsbuild]

Pull request #23895 was updated. @perrotta, @cmsbuild, @civanch, @mdhildreth, @slava77 can you please check and sign again.

[perrotta]

@cmsbuild please test
(@smuzaffar : thank you for setting up the 76X IB!)

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30133/console Started: 2018/08/29 23:24

[cmsbuild]

Pull request #23895 was updated. @perrotta, @cmsbuild, @civanch, @mdhildreth, @slava77 can you please check and sign again.

[perrotta]

please test

[cmsbuild]

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30310/console Started: 2018/09/10 10:14

[cmsbuild]

+1
Tested at: 4e9c2f2
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23895/30310/summary.html

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-23895/30310/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 48 differences found in the comparisons
• DQMHistoTests: Total files compared: 14
• DQMHistoTests: Total histograms compared: 1009343
• DQMHistoTests: Total failures: 530
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 1008345
• DQMHistoTests: Total skipped: 468
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -13 KiB( 13 files compared)
• Checked 56 log files, 4 edm output root files, 14 DQM output files

[perrotta]

There are differences in the comparisons.

They were already there also before the last revert, see #23895 (comment): my fault having overlooked them.

They must be explained before we can proceed with the backport. And one should also investigate why those difference show up only here in 76X (quite a lot of workflows) and not in 80X.

Please @hvanhaev have a look

[hvanhaev]

Hi Andrea, Is there a twiki page or so where I can figure out to what comparison to look at? If I click on “Comparison with the baseline” I get a very cryptic list of directories and I have no idea where to look at. If I open one random directory, e.g. 25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/ I see that there are e.f. a lot of failures in DQM, but none in Castor… If someone can point me to a manual on how to interpret all this that would help me a lot. Thanks, Hans On 10 Sep 2018, at 12:32, perrotta <notifications@github.com<mailto:notifications@github.com>> wrote: There are differences in the comparisons. They were already there also before the last revert, see #23895 (comment)<#23895 (comment)>: my fault having overlooked them. They must be explained before we can proceed with the backport. And one should also investigate why those difference show up only here in 76X (quite a lot of workflows) and not in 80X. Please @hvanhaev<https://github.com/hvanhaev> have a look — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#23895 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEgkF8PDhBXo169f3hz4X9w7Z87kNFDbks5uZj-rgaJpZM4VX7Dv>.

[perrotta]

Hi Hans.

You can scroll down till the bottom of the page linked by "Comparison with the baseline”, which shows the results of the DQM comparisons, and go down to "validateJR". That page shows the comparisons for reco quantities (direct link: https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_6_X_2018-09-09-0000+23895/28359/validateJR/)

One of the files in that page is "logRootQA.log": if you open it, towards its bottom you find the following list

/build/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison
JR results differ 4 all_OldVSNew_RunMinBias2011Awf1000p0
JR results differ 6 all_OldVSNew_ZMM13TeVwf1330p0
JR results differ 4 all_OldVSNew_RunSinglePh2015Dwf134p911
JR results differ 7 all_OldVSNew_Higgs200ChargedTauswf9p0
JR results differ 6 all_OldVSNew_RunMinBias2011Awf1001p0
JR results differ 7 all_OldVSNew_TTbarFSwf5p1
JR results differ 2 all_OldVSNew_RunHI2011wf140p53
JR results differ 3 all_OldVSNew_TTbarwf25p0
JR results differ 2 all_OldVSNew_RunCosmics2011Awf4p22
JR results differ 1 all_OldVSNew_ZEEFS13wf135p4
JR results differ 6 all_OldVSNew_TTbarPUwf25202p0
SUMMARY Reco comparison results: 48 differences found in the comparisons

which tells you which workflows show differences.

You can access, any of them, e.g. "all_OldVSNew_ZMM13TeVwf1330p0":
https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_7_6_X_2018-09-09-0000+23895/28359/validateJR/all_OldVSNew_ZMM13TeVwf1330p0/

[perrotta]

@hvanhaev @fabiocos

Slava reminded me this old issue that was fixed since 8_0_X, but never backported to 7_6_X:
#13506

What we see here in the jenkins comparisons exactly corresponds to what was observed at the time, and has nothing to do with Castor in any case. That is therefore the origin of the (kind of random) differences observed, and it is confirmed that this pull request do not change the outputs with the default config which is implemented.

[perrotta]

+1

• Correct backport of the code needed to apply fixes for the analysis and reprocessings of Castor data, which do not change the previous behaviour in its default configuration: the last addition since previous reco signature just reinforced it.

[hvanhaev]

Dear Andrea, Thanks a lot for the information regarding the comparisons (now I know where to look at) and the actual reason for the differences. Cheers, Hans On 10 Sep 2018, at 16:21, perrotta <notifications@github.com<mailto:notifications@github.com>> wrote: +1 * Correct backport of the code needed to apply fixes for the analysis and reprocessings of Castor data, which do not change the previous behaviour in its default configuration: the last addition since previous reco signature just reinforced it. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub<#23895 (comment)>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AEgkFyCRexVUSxqPDVU1JQ8vDyB5PxQkks5uZnVUgaJpZM4VX7Dv>.

[fabiocos]

+1

[fabiocos]

merge

@civanch you already signed, now the behaviour is fully backward compatible, in case please sign it again for reference