Make new HF Shower Library the default for Run3

[abdoulline]

PR description:

Presentation at Simulation meeting on July 16 (updated on July 26)
https://indico.cern.ch/event/1060045/contributions/4455234/attachments/2282906/3888021/Status_newHF_SL_S.A.July16_2021.pdf

Uses earlier implemented (in preparatory PR #34432) modifier "run3_HFSL".
The library file itself has been added to cmsdist several days ago cms-sw/cmsdist#7162
In addtion, standalone HF Shower Library assembly code/config is updated in this PR.

PR validation:

Some results available in the aforementioned presentation.
Technical test:
runTheMatrix.py -l limited
OK

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-34626/24212

• This PR adds an extra 24KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/StandardSequences/python/Eras.py modified in PR(s): DRAFT: nano: adaptation to run in Prompt #34329, Add run3 tracking low pu era #33532
  • File SimCalorimetry/HcalSimProducers/python/hcalSimParameters_cfi.py modified in PR(s): TDC simulation for QIE10 (HF) #33829

[cmsbuild]

A new Pull Request was created by @abdoulline (Salavat Abdullin) for master.

It involves the following packages:

• Configuration/Eras (operations)
• Configuration/StandardSequences (operations)
• SimCalorimetry/HcalSimProducers (simulation)
• SimG4CMS/ShowerLibraryProducer (simulation)

@civanch, @silviodonato, @mdhildreth, @cmsbuild, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@fabiocos, @makortel, @felicepantaleo, @GiacomoSguazzoni, @JanFSchulte, @rovere, @VinInn, @Martin-Grunewald, @bsunanda, @lecriste, @cvuosalo, @mtosi, @ebrondol, @mmusich, @dgulhan, @slomeo, @mariadalfonso this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy, @perrotta you are the release manager for this.

cms-bot commands are listed here

[civanch]

please test

[civanch]

@abdoulline , I have started tests having in mind that something more will be required probably for this PR.

[abdoulline]

@civanch thank you.
Not sure what else is needed (?).

[abdoulline]

Looking at the console
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-relvals/8137/console
I have the impression that many wf's failed at step1 (exit 31744) failing to get access to INPUT,
while in my case (runTheMatrix.py -l limited ) it was all OK.
And it seems to be a common problem for several PRs tonight...

[smuzaffar]

looks like there is issue with network or EOS, many relval-INPUT tests are failing for other PRs too

[smuzaffar]

issue is related to #34633 . For Relval-INPUT tests we timeout the job after 15mins and looks like these failed wf are taking more than 15 mins to start and then killed by timeout

[cmsbuild]

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f933d/17208/summary.html
COMMIT: 3a93db7
CMSSW: CMSSW_12_0_X_2021-07-26-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34626/17208/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

• 11601.011601.0_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt10+2021+SingleElectronPt10_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
• 11602.011602.0_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt35+2021+SingleElectronPt35_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log
• 11603.011603.0_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA/step2_SingleElectronPt1000+2021+SingleElectronPt1000_pythia8_GenSimINPUT+Digi+Reco+HARVEST+ALCA.log

Expand to see more relval errors ...

• 11604.0
• 11605.0
• 11606.0
• 11607.0
• 11608.0
• 11609.0
• 11630.0
• 11634.0
• 11634.1
• 11634.24
• 11634.5
• 11634.501
• 11634.505
• 11646.0
• 11723.17
• 11634.511
• 11634.521
• 11634.7
• 11634.91
• 11640.0
• 11643.0
• 11650.0
• 11650.501
• 11650.505
• 11725.0
• 11834.0
• 11834.13
• 11834.24
• 11846.0
• 11925.0
• 12034.0
• 12434.0
• 12634.0
• 12834.0
• 13034.0
• 23434.21
• 23434.99
• 23434.9921
• 34834.21
• 34834.9921

Comparison Summary

Summary:

• No significant changes to the logs found
• Reco comparison results: 27199 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 242767
• DQMHistoTests: Total nulls: 229
• DQMHistoTests: Total successes: 2755546
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.004 KiB( 38 files compared)
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: found differences in 6 / 38 workflows

[civanch]

+1

test problem is not connected with this PR.
@smuzaffar , @qliphy , should we restart the test or this can be merged?

[smuzaffar]

yes errors are not due to this PR. No need to start the full tests, I have restart only the relval-INPUT with higher timeout. Hopefully this will allow to let the test finish properly

[civanch]

urgent

[civanch]

Please, note, that this PR change simulation histories.

[civanch]

@qliphy, this PR is good for pre5 from our point of view.

[qliphy]

+operations

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy, @perrotta (and backports should be raised in the release meeting by the corresponding L2)

[qliphy]

please test
(Just to test again. As #34638 is merged, which should remove the RelVals-INPUT errors)

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8f933d/17279/summary.html
COMMIT: 3a93db7
CMSSW: CMSSW_12_0_X_2021-07-27-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/34626/17279/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

The workflows 140.53 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

Summary:

• No significant changes to the logs found
• Reco comparison results: 29406 differences found in the comparisons
• DQMHistoTests: Total files compared: 39
• DQMHistoTests: Total histograms compared: 2998564
• DQMHistoTests: Total failures: 246432
• DQMHistoTests: Total nulls: 248
• DQMHistoTests: Total successes: 2751862
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -45.699 KiB( 38 files compared)
• DQMHistoSizes: changed ( 140.53 ): -44.531 KiB Hcal/DigiRunHarvesting
• DQMHistoSizes: changed ( 140.53 ): -1.172 KiB RPC/DCSInfo
• DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
• Checked 165 log files, 37 edm output root files, 39 DQM output files
• TriggerResults: found differences in 6 / 38 workflows

[abdoulline]

Except for wf 140.53 (different files in step1_dasquery.log) changes are observed in 2021/2023/2026 wfs as expected. And (given limited statistics) they all look reasonable.

[qliphy]

+1