SiStripHitEfficiency workflow in the Prompt Calibration Loop

[mmusich]

PR description:

This is a follow-up to PR #37530, and applies the necessary change-set to be able to run the workflow sketched in that PR as part of the Prompt Calibration Loop.
I also updated various unit tests and relval workflows to be able to test the new addition.

PR validation:

I've run the following commands to test the ALCAPROMPT production:

cmsDriver.py testReAlCa -s ALCA:PromptCalibProdSiStripHitEfficiency --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --datatier ALCARECO --eventcontent ALCARECO --processName=ReAlCa -n 100000 --dasquery='file dataset=/StreamExpress/Run2018D-SiStripCalMinBias-Express-v1/ALCARECO run=325172' --nThreads=4 --no_exec 

followed by:

cmsDriver.py stepHarvest -s ALCAHARVEST:SiStripHitEff --conditions 123X_dataRun2_v2 --scenario pp --data --era Run2_2018 --filein file:PromptCalibProdSiStripHitEfficiency.root -n -1 --no_exec                        

to test the ALCAHARVESTING step.
I have also run successfully:

• runTheMatrix.py -l 1001.0, 1001.2
• scram b runtests

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport, but a backport to 12_3_X for usage in first beams might be desirable.

cc:
@robervalwalsh @mdelcourt @sroychow

[tvami]

test parameters:

• workflows = 1001.0, 1001.2

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-37708/29536

• This PR adds an extra 92KB to repository

[cmsbuild]

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

• CalibTracker/SiStripHitEfficiency (alca)
• Calibration/TkAlCaRecoProducers (alca)
• Configuration/AlCa (alca)
• Configuration/EventContent (operations)
• Configuration/PyReleaseValidation (pdmv, upgrade)
• Configuration/StandardSequences (operations)

@perrotta, @malbouis, @yuanchao, @jordan-martins, @bbilin, @tvami, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @qliphy, @francescobrivio, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@echabert, @felicepantaleo, @pieterdavid, @robervalwalsh, @kpedro88, @Martin-Grunewald, @trtomei, @threus, @mmusich, @slomeo, @makortel, @JanFSchulte, @dgulhan, @missirol, @beaucero, @GiacomoSguazzoni, @rovere, @VinInn, @tocheng, @ebrondol, @mtosi, @fabiocos, @gbenelli, @lecriste this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

[mmusich]

@cmsbuild, please test

• though I expect 1001.2 to fail, see ProductNotFound exception in AlcaPCCIntegrator in workflow 1001.2 #37685

[tvami]

@mmusich please edit the "workflow_threading = 1001.0 1001.2" to "workflow_threading = 1001.0,1001.2" I see now there is an error

[mmusich]

@tvami, I edited 1h ago...

[tvami]

why is it thumbs-down?

[smuzaffar]

test parameters:

• enable_test = threading
• workflow_threading = 1001.0

[smuzaffar]

bot is failing to change delete the thumbs down. So I have deleted the original comment and added a new one

[smuzaffar]

please test

[tvami]

@cmsbuild , please test with #37712

[mmusich]

@cmsbuild please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2c516b/24441/summary.html
COMMIT: 5e79159
CMSSW: CMSSW_12_4_X_2022-05-04-1100/slc7_amd64_gcc10
Additional Tests: THREADING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/37708/24441/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
39434.501 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

The workflows 1001.0 have different files in step1_dasquery.log than the ones found in the baseline. You may want to check and retrigger the tests if necessary. You can check it in the "files" directory in the results of the comparisons

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-2c516b/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA

Summary:

• No significant changes to the logs found
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 50
• DQMHistoTests: Total histograms compared: 3700590
• DQMHistoTests: Total failures: 7
• DQMHistoTests: Total nulls: 1
• DQMHistoTests: Total successes: 3700560
• DQMHistoTests: Total skipped: 22
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: -0.004 KiB( 48 files compared)
• DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
• Checked 200 log files, 45 edm output root files, 50 DQM output files
• TriggerResults: no differences found

[mmusich]

urgent

• just to flag it's necessary for the next pre-release.

[tvami]

+alca

• tests (threaded included) https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2c516b/24441/runTheMatrixTHREADING-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5+ALCAHARVD7+ALCAHARVD8/ and https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2c516b/24441/runTheMatrixTHREADING-results/1001.2_RunZeroBias2017F+RunZeroBias2017F+TIER0EXPRUN2+ALCAEXPRUN2+ALCAHARVDSIPIXELCAL+ALCAHARVDSIPIXELCALLA/ run fine

[tvami]

@cms-sw/upgrade-l2 @cms-sw/pdmv-l2 would you please re-sign this PR? thanks!

[kskovpen]

+pdmv

[srimanob]

+Upgrade

resign.

[tvami]

@cms-sw/orp-l2 this is essentially fully signed now!

[perrotta]

+1

• Thank you @mmusich

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.