AlCa DQM sequence for express stream #35850

jfernan2 · 2021-10-26T16:08:25Z

PR description:

Added @express to Express pp AlCas and to UnitTest Configuration/DataProcessing/test/run_CfgTest.sh

Express AlCaRecos were taking as default DQM sequence @standardDQM which is not appropiated since not all products are available in reconstruction from AlCaReco stream. @express contains a subset of DQM sequences which fits better the AlCa streams

The corner cases for HeavyIons and Cosmics, in principle are running by default reduced sequences driven by HI and Cosmics scenarios, so the default DQM sequence seems to work and no dedicated @expressHI or @expressCosmics are needed (see below for Validation of this statement, up to GlobalTag settings). For a full validation I would need advice from @tvami about which GTs to use for Cosmics and HI

This PR solves issue #35586

Following changes suggested by @mmusich (thanks a lot!!) in:
#35586 (comment)

I understand this PR should go hand by hand with changes in Tier0 configuration for future processings setting @express there, in particular :

https://github.com/dmwm/T0/blob/master/etc/ProdOfflineConfiguration.py#L227 no DQM sequences mentioned explicitly
in https://github.com/dmwm/T0/blob/master/src/python/T0/RunConfig/Tier0Config.py#L890 initialized to an empty sequence

FYI: @mmusich @slava77 @tvami @jpata

PR validation:

Tested that Configuration/DataProcessing/test/run_CfgTest.sh creates valid python configurations
HI case: tested that it runs up to GlobalTag needs the following command

python3 Configuration/DataProcessing/test/RunExpressProcessing.py --scenario HeavyIonsEra_Run2_2018 --global-tag 121X_mcRun3_2021_realistic_v11Based_forExpressRateStudies --lfn file:./F4CD260A-B956-084B-AFC3-EBFFB2A4E177.root --fevt --dqmio --alcareco TkAlMinBiasHI+SiStripCalMinBias

where input file came from: xrdcp root://cms-xrd-global.cern.ch//store/hidata/HIRun2018A/HIDoubleMuon/RAW/v1/000/327/554/00000/F4CD260A-B956-084B-AFC3-EBFFB2A4E177.root .

Cosmic case: tested that it runs up to GlobalTag needs the following command

python3 Configuration/DataProcessing/test/RunExpressProcessing.py --scenario cosmicsEra_Run2_2018 --global-tag 121X_mcRun3_2021_realistic_v11Based_forExpressRateStudies --lfn file:./eaceff12-9817-41eb-bc10-c133fbb62bf2.root --fevt --dqmio --alcareco SiStripCalCosmicsNano

where input file came from: xrdcp root://cms-xrd-global.cern.ch//store/data/Commissioning2021/Cosmics/RAW/v1/000/345/823/00000/eaceff12-9817-41eb-bc10-c133fbb62bf2.root .

jfernan2 · 2021-10-26T16:08:43Z

please test

cmsbuild · 2021-10-26T16:17:25Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-35850/26223

This PR adds an extra 12KB to repository

cmsbuild · 2021-10-26T16:17:46Z

A new Pull Request was created by @jfernan2 for master.

It involves the following packages:

Configuration/DataProcessing (operations)

@perrotta, @qliphy, @fabiocos, @davidlange6 can you please review it and eventually sign? Thanks.
@Martin-Grunewald, @missirol, @fabiocos this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2021-10-26T19:57:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eea0af/19963/summary.html
COMMIT: 2447664
CMSSW: CMSSW_12_1_X_2021-10-26-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35850/19963/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 41
DQMHistoTests: Total histograms compared: 2797338
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2797310
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 40 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 173 log files, 37 edm output root files, 41 DQM output files
TriggerResults: no differences found

perrotta · 2021-10-27T09:14:15Z

Test scripts are reported to work as expected in the PR description
@jpata can you confirm that this PR fixes the problem reported in your issue #35586?

tvami · 2021-10-27T09:20:57Z

Hi @jfernan2 please use the auto phase1_2021_cosmics and phase1_2021_realistic_hi keys for testing the cosmics and HI respectively. On the other hand you'll have issues with the input file, the whose issue #35586 is for MC. I'm going to comment that in that thread.

jfernan2 · 2021-10-27T14:52:00Z

Thanks a lot @tvami
However, I tried them and for both GTs I get:
Global Tag "phase1_2021_realistic_hi" has not been found in the database. from GTProxy::load
Global Tag "phase1_2021_cosmics" has not been found in the database. from GTProxy::load

I am using: CMSSW_12_1_X_2021-10-25-2300
What do I miss?
Thanks

tvami · 2021-10-27T15:07:26Z

phase1_2021_realistic_hi

the auto part of it. The is the literal version of what you need
--conditions auto:phase1_2021_realistic_hi
and
--conditions auto:phase1_2021_cosmics

jfernan2 · 2021-10-27T15:23:11Z

Hi @tvami
The thing is that --conditions is not accepted by RunExpressProcessing.py

If I try with --global-tag I get:
Exception: no correspondence for phase1_2021_cosmics,frontier://PromptProd/CMS_CONDITIONS
Exception: no correspondence for phase1_2021_cosmics,frontier://PromptProd/CMS_CONDITIONS

I attach here what I do:
error.txt

Thanks

tvami · 2021-10-27T15:58:47Z

Hi @tvami The thing is that --conditions is not accepted by RunExpressProcessing.py

If I try with --global-tag I get: Exception: no correspondence for phase1_2021_cosmics,frontier://PromptProd/CMS_CONDITIONS Exception: no correspondence for phase1_2021_cosmics,frontier://PromptProd/CMS_CONDITIONS

I attach here what I do: error.txt

Thanks

Ahh I didn't know that, in that case please use 121X_mcRun3_2021_realistic_HI_v15 and 121X_upgrade2018cosmics_realistic_deco_v9 sorry for not saying this on the first place

jfernan2 · 2021-10-27T16:40:51Z

Thanks @tvami !
With those GTs I validate HI, for Cosmics I get an abort signal from Module: HBHEPhase1Reconstructor:hbhereco
cmsRun: /data/cmsbld/jenkins/workspace/build-any-ib/w/tmp/BUILDROOT/5d9867eb8cb052ef99563d12a74bf73b/opt/cmssw/slc7_amd64_gcc900/cms/cmssw/CMSSW_12_1_X_2021-10-25-2300/src/CalibCalorimetry/HcalAlgos/interface/HcalSiPMnonlinearity.h:10: HcalSiPMnonlinearity::HcalSiPMnonlinearity(const std::vector&): Assertion `pars.size() == 3' failed.

python3 RunExpressProcessing.py --scenario cosmicsEra_Run2_2018 --global-tag 121X_upgrade2018cosmics_realistic_deco_v9 --lfn file:./eaceff12-9817-41eb-bc10-c133fbb62bf2.root --fevt --dqmio --alcareco SiStripCalCosmicsNano
Retrieved Scenario: cosmicsEra_Run2_2018
Using Global Tag: 121X_upgrade2018cosmics_realistic_deco_v9
Configuring to Write out FEVT
Configuring to Write out DQMIO
Configuring to Write out ALCARECO
removing ENDJOB from steps since not compatible with DQMIO dataTier
^[[A^[[A^[[A
entry tobeoverwritten.xyz
Step: RAW2DIGI Spec:
Step: L1Reco Spec:
Step: RECO Spec:
Step: ALCAPRODUCER Spec: ['SiStripCalCosmicsNano']
Step: DQM Spec:
customising the process with customiseCosmicData from Configuration/DataProcessing/RecoTLR
Now do:
cmsRun -e RunExpressProcessingCfg.py

tvami · 2021-10-27T16:52:06Z

I don't know if we expect this to work. Running on data with an MC GT doesn't seem like something that we expect to work... I guess for a real validation one needs to create HI and Cosmics MCs. I dont think we are in a hurry anymore as the purpose of the original exercise is irrelevant now that we already have collisions (I mean that originally we wanted to study the rates, now one can just look at the relevant plot for data in today's datataking, right?)

jfernan2 · 2021-10-27T16:55:42Z

I agree. I did the exercise since you were asking for Cosmics and HI corner cases.

mmusich · 2021-10-27T17:35:40Z

hold

cmsbuild · 2021-10-27T17:36:01Z

Pull request has been put on hold by @mmusich
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

mmusich · 2021-11-30T10:17:12Z

the problem with:

Exception Message:
HLTHighLevel [instance: ALCARECOSiStripCalMinBiasAAGHLT - path: pathALCARECOSiStripCalMinBiasAAG]: No triggerList with key SiStripCalMinBiasAAG in AlCaRecoTriggerBitsRcd
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 30-Nov-2021 10:06:41 CET-----------------------
An exception of category 'Configuration' occurred while
[0] Processing Event run: 1 lumi: 724 event: 72303 stream: 3
[1] Running path 'pathALCARECOLumiPixelsMinBias'
[2] Calling method for module HLTHighLevel/'ALCARECOLumiPixelsMinBiasHLT'
Exception Message:
HLTHighLevel [instance: ALCARECOLumiPixelsMinBiasHLT - path: pathALCARECOLumiPixelsMinBias]: No triggerList with key LumiPixelsMinBias in AlCaRecoTriggerBitsRcd
----- End Fatal Exception -------------------------------------------------
----- Begin Fatal Exception 30-Nov-2021 10:06:41 CET-----------------------
An exception of category 'Configuration' occurred while
[0] Processing Event run: 1 lumi: 724 event: 72307 stream: 4
[1] Running path 'pathALCARECOSiStripCalMinBiasAAG'
[2] Calling method for module HLTHighLevel/'ALCARECOSiStripCalMinBiasAAGHLT'
Exception Message:
HLTHighLevel [instance: ALCARECOSiStripCalMinBiasAAGHLT - path: pathALCARECOSiStripCalMinBiasAAG]: No triggerList with key SiStripCalMinBiasAAG in AlCaRecoTriggerBitsRcd

is exactly the same thing that is described at #35586 (comment)
That's (again) because there are not the appropriate AlCaRecoTriggerBitsRcd payloads in the HI GlobalTag (121X_mcRun3_2021_realistic_HI_v15)

tvami · 2021-11-30T15:38:11Z

Hi @jfernan2 I've created the new GTs

for HI: 121X_mcRun3_2021_realistic_HI_forExpressRateStudies
diff: https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021_realistic_HI_v15/121X_mcRun3_2021_realistic_HI_forExpressRateStudies
(also has relevance zero material changes, can be ignored)
for cosmics: 121X_mcRun3_2021cosmics_realistic_deco_forExpressRateStudies
diff: https://cms-conddb.cern.ch/cmsDbBrowser/diff/Prod/gts/121X_mcRun3_2021cosmics_realistic_deco_v15/121X_mcRun3_2021cosmics_realistic_deco_forExpressRateStudies
(same conclusion)

Please try with these, thanks!

jfernan2 · 2021-11-30T15:59:33Z

Thanks @tvami I confirm the first one of the two new GTs works well with the MinBias MC file, despite they are 12_1_X based and I run over 12_2_0_pre2. The command I did on top of this PR, for the records, was:

python3 Configuration/DataProcessing/test/RunExpressProcessing.py --nThreads 8 --scenario ppEra_Run3 --global-tag 121X_mcRun3_2021_realistic_HI_forExpressRateStudies --lfn file:./12d0db89-7974-4418-bb7e-1e2fbb548d2e.root --fevt --dqmio --alcarecos SiStripPCLHistos+SiStripCalZeroBias+SiStripCalMinBias+SiStripCalMinBiasAAG+TkAlMinBias+LumiPixelsMinBias+SiPixelCalZeroBias+PromptCalibProd+PromptCalibProdSiStrip+PromptCalibProdSiPixelAli+PromptCalibProdSiStripGains+PromptCalibProdSiStripGainsAAG+PromptCalibProdSiPixel --dqmSeq=@express

With the second (Cosmics) GT I get a crash, I guess that due to the fact that I do not have a MC Cosmic file:
An exception of category 'NoProxyException' occurred while
[0] Processing Event run: 1 lumi: 724 event: 72308 stream: 7
[1] Running path 'dqmoffline_8_step'
[2] Prefetching for module LogMessageMonitor/'TrackFinderLogMessageMonMB'
[3] Prefetching for module LogErrorHarvester/'logErrorHarvester'
[4] Prefetching for module RecoTauDiscriminantCutMultiplexer/'hpsPFTauDiscriminationByMVA6ElectronRejection'
[5] Calling method for module PFRecoTauDiscriminationAgainstElectronMVA6/'hpsPFTauDiscriminationByMVA6rawElectronRejection'
Exception Message:
No data of type "GBRForest" with label "RecoTauTag_antiElectronMVA_NoEleMatch_woGwoGSF_BL" in record "GBRWrapperRcd"

tvami · 2021-11-30T16:39:25Z

I do not have a MC Cosmic file

Yes, I think that's the issue indeed.

mmusich · 2021-11-30T17:00:11Z

Yes, I think that's the issue indeed.

one can generate it with one of the 7.xx wfs

$ runTheMatrix.py --show | grep Cosmics | grep '7.'
7.0    Cosmics+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS 
7.1    CosmicsSPLoose+DIGICOS+RECOCOS+ALCACOS+HARVESTCOS 
7.2    Cosmics_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18 
7.21   Cosmics_UP17+DIGICOS_UP17+RECOCOS_UP17+ALCACOS_UP17+HARVESTCOS_UP17 
7.22   Cosmics_UP16+DIGICOS_UP16+RECOCOS_UP16+ALCACOS_UP16+HARVESTCOS_UP16 
7.23   Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21 
7.24   Cosmics_UP21_0T+DIGICOS_UP21_0T+RECOCOS_UP21_0T+ALCACOS_UP21_0T+HARVESTCOS_UP21_0T 
7.3    CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18 
7.4    Cosmics_UP18+DIGICOSPEAK_UP18+RECOCOSPEAK_UP18+ALCACOS_UP18+HARVESTCOS_UP18

tvami · 2021-11-30T17:21:12Z

Hi @jfernan2 I very quickly created the RAW file for you for wf 7.23
You can find the file in
/afs/cern.ch/user/t/tvami/public/PRsToCMSSW/MCExpressWf/CMSSW_12_1_0/src/step2_DIGI_L1_DIGI2RAW_HLT.root

jfernan2 · 2021-11-30T17:38:03Z

Thanks @mmusich @tvami
I have tried with 7.23 and it runs succesfully:

python3 Configuration/DataProcessing/test/RunExpressProcessing.py --nThreads 8 --scenario cosmics --global-tag 121X_mcRun3_2021cosmics_realistic_deco_forExpressRateStudies --lfn file:./7.23_Cosmics_UP21+Cosmics_UP21+DIGICOS_UP21+RECOCOS_UP21+ALCACOS_UP21+HARVESTCOS_UP21/step2.root --fevt --dqmio --alcarecos=TkAlCosmics0T+SiStripCalZeroBias

Please note that DQM sequence should be specified in case of MC Minbias/Data (@express) since none of the config files modified in this PR have a switch for mc case as cmsDriver.py does.

In the case of Cosmics MC, since DQM does not distinguish between Real or Simulated Cosmics, the above statement does not apply.

If not DQM sequence is specified, @standardDQM is used by default as explained in #35586 (comment) and comments onwards.
Thanks

mmusich · 2021-11-30T17:41:22Z

unhold

@tvami it would be perhaps convenient for future checks to include the AlCaRecoTriggerBitsRcs updates that you have proposed here #35850 (comment) and here #35586 (comment) in the autoCond GTs.

tvami · 2021-11-30T17:43:55Z

Hi @mmusich hmm ok, I just thought all this was for a special case, and we dont really need to have it in autoCond for all RelVals/etc

mmusich · 2021-11-30T17:45:10Z

just thought all this was for a special case, and we dont really need to have it in autoCond for all RelVals/etc

well, in that case you'll have to redo special GTs by hand in all future occurrences of such checks. Please state if you see any con in making this the default.

perrotta · 2021-12-22T13:31:27Z

@jfernan2 @tvami @mmusich do I understand correctly that we can consider this PR ready for getting merged?
Was there any progress on the proposal of including the AlCaRecoTriggerBitsRcs updates in the autoCond GTs?

perrotta · 2021-12-22T13:31:39Z

please test

cmsbuild · 2021-12-22T17:34:06Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-eea0af/21432/summary.html
COMMIT: 2447664
CMSSW: CMSSW_12_3_X_2021-12-21-2300/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/35850/21432/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 43
DQMHistoTests: Total histograms compared: 3461692
DQMHistoTests: Total failures: 5
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3461665
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 42 files compared)
Checked 181 log files, 42 edm output root files, 43 DQM output files
TriggerResults: no differences found

tvami · 2021-12-23T15:15:46Z

Was there any progress on the proposal of including the AlCaRecoTriggerBitsRcs updates in the autoCond GTs?

Yes we did that, this is the PR #36326
and this is the HN msg: https://hypernews.cern.ch/HyperNews/CMS/get/calibrations/4485/2/1/1/1.html

perrotta · 2021-12-29T12:53:00Z

"So, I guess I should close too the PR I made for fixing this issue which was put on hold: #35850", see
#35586 (comment)

Pending further clarifications, we'll keep this PR on hold

perrotta · 2021-12-29T12:53:06Z

hold

cmsbuild · 2021-12-29T12:53:24Z

Pull request has been put on hold by @perrotta
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

perrotta · 2022-02-24T06:53:00Z

@jfernan2 @tvami (periodic check): can this PR get closed, given #35586 (comment)? Or any other action is expected, also in the future?

jfernan2 · 2022-02-24T09:20:32Z

@perrotta I don't have objections as I have expressed before. Let me know if I should close it please. Thanks

perrotta · 2022-02-24T10:22:55Z

@perrotta I don't have objections as I have expressed before. Let me know if I should close it please. Thanks

Thank you @jfernan2
You opened it, and you are supposed to close it as soon as it is not considered useful any more.
However, as far as I can understand everybody agrees on closing it, so that anyone can be considered allowed to do so: I'm closing it now, then.

Added @express to Express pp AlCas and to UnitTest

2447664

cmsbuild added this to the CMSSW_12_1_X milestone Oct 26, 2021

cmsbuild added code-checks-pending operations-pending orp-pending pending-signatures tests-started labels Oct 26, 2021

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 26, 2021

cmsbuild added tests-approved and removed tests-started labels Oct 26, 2021

cmsbuild added the hold label Oct 27, 2021

smuzaffar modified the milestones: CMSSW_12_1_X, CMSSW_12_2_X Oct 29, 2021

jfernan2 mentioned this pull request Nov 5, 2021

Issues with ALCA and DQM modules on the MC express stream #35586

Closed

cmsbuild removed the hold label Nov 30, 2021

smuzaffar modified the milestones: CMSSW_12_2_X, CMSSW_12_3_X Dec 6, 2021

cmsbuild added tests-started and removed tests-approved labels Dec 22, 2021

cmsbuild added tests-approved and removed tests-started labels Dec 22, 2021

cmsbuild added the hold label Dec 29, 2021

perrotta closed this Feb 24, 2022

mmusich mentioned this pull request Sep 14, 2022

AlCa Run3 Heavy Ion scenarios #39387

Merged

AlCa DQM sequence for express stream #35850

AlCa DQM sequence for express stream #35850

Conversation

jfernan2 commented Oct 26, 2021

PR description:

PR validation:

jfernan2 commented Oct 26, 2021

cmsbuild commented Oct 26, 2021

cmsbuild commented Oct 26, 2021

cmsbuild commented Oct 26, 2021

Comparison Summary

perrotta commented Oct 27, 2021

tvami commented Oct 27, 2021

jfernan2 commented Oct 27, 2021

tvami commented Oct 27, 2021

jfernan2 commented Oct 27, 2021

tvami commented Oct 27, 2021

jfernan2 commented Oct 27, 2021

tvami commented Oct 27, 2021

jfernan2 commented Oct 27, 2021

mmusich commented Oct 27, 2021

cmsbuild commented Oct 27, 2021

mmusich commented Nov 30, 2021

tvami commented Nov 30, 2021

jfernan2 commented Nov 30, 2021

tvami commented Nov 30, 2021

mmusich commented Nov 30, 2021

tvami commented Nov 30, 2021

jfernan2 commented Nov 30, 2021

mmusich commented Nov 30, 2021

tvami commented Nov 30, 2021

mmusich commented Nov 30, 2021

perrotta commented Dec 22, 2021

perrotta commented Dec 22, 2021

cmsbuild commented Dec 22, 2021

Comparison Summary

tvami commented Dec 23, 2021

perrotta commented Dec 29, 2021

perrotta commented Dec 29, 2021

cmsbuild commented Dec 29, 2021

perrotta commented Feb 24, 2022

jfernan2 commented Feb 24, 2022

perrotta commented Feb 24, 2022