More plots from GE21 for the GEM DQMs #38398

quark2 · 2022-06-16T20:44:06Z

PR description:

We have several updates on GEM DQMs.

Migrations of some plots to other places (online -> offline or vice versa)
More plots for GE21 is now available

PR validation:

Test are done and one can check again by runTheMatrix workflows

@jshlee @watson-ij @seungjin-yang

cmsbuild · 2022-06-16T20:51:20Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38398/30598

This PR adds an extra 40KB to repository

cmsbuild · 2022-06-16T20:51:41Z

A new Pull Request was created by @quark2 for master.

It involves the following packages:

DQM/GEM (dqm)

@emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @micsucmed, @rvenditti can you please review it and eventually sign? Thanks.
@watson-ij this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

jshlee · 2022-06-16T20:52:15Z

please test

cmsbuild · 2022-06-16T22:04:10Z

-1

Failed Tests: UnitTests RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b4b802/25574/summary.html
COMMIT: a262ae5
CMSSW: CMSSW_12_5_X_2022-06-16-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38398/25574/install.sh to create a dev area with all the needed externals and cmssw changes.

Unit Tests

I found errors in the following unit tests:

---> test TestDQMOfflineConfiguration_110 had ERRORS
---> test TestDQMOfflineConfiguration_160 had ERRORS
---> test TestDQMOfflineConfiguration_0 had ERRORS
---> test TestDQMOfflineConfiguration_190 had ERRORS
and more ...

RelVals

136.7611136.7611_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM/step2_RunJetHT2016E_reminiaod+RunJetHT2016E_reminiaod+REMINIAOD_data2016_HIPM+HARVESTDR2_REMINIAOD_data2016_HIPM.log
136.8523136.8523_RunJetHT2018C_nanoULremini+RunJetHT2018C_nanoULremini+NANOEDM2018_106Xv2+HARVESTNANOAOD2018_106Xv2/step2_RunJetHT2018C_nanoULremini+RunJetHT2018C_nanoULremini+NANOEDM2018_106Xv2+HARVESTNANOAOD2018_106Xv2.log
136.8311136.8311_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017/step2_RunJetHT2017F_reminiaod+RunJetHT2017F_reminiaod+REMINIAOD_data2017+HARVEST2017_REMINIAOD_data2017.log

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
136.72411136.72411_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMINIAOD_data2016UL_HIPM+HARVESTDR2_REMINIAOD_data2016UL_HIPM.log
136.72412136.72412_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM/step2_RunJetHT2016B_reminiaodUL+RunJetHT2016B_reminiaodUL+REMININANO_data2016UL_HIPM+HARVESTDR2_REMININANO_data2016UL_HIPM.log

Expand to see more relval errors ...

cmsbuild · 2022-06-17T04:38:08Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38398/30601

This PR adds an extra 40KB to repository

cmsbuild · 2022-06-17T04:38:34Z

Pull request #38398 was updated. @emanueleusai, @ahmad3213, @cmsbuild, @jfernan2, @pmandrik, @micsucmed, @rvenditti can you please check and sign again.

jshlee · 2022-06-17T12:13:09Z

please test

cmsbuild · 2022-06-17T14:48:41Z

-1

Failed Tests: RelVals RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b4b802/25591/summary.html
COMMIT: aa3a8f5
CMSSW: CMSSW_12_5_X_2022-06-17-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38398/25591/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

138.4

----- Begin Fatal Exception 17-Jun-2022 14:36:30 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module GEMDAQStatusSource/'GEMDAQStatusSource'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: MuonDigiCollection<unsigned short,GEMAMC13Status>
Looking for module label: muonGEMDigis
Looking for productInstanceName: AMC13Status

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

138.5

----- Begin Fatal Exception 17-Jun-2022 14:39:26 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 346512 lumi: 250 event: 243042266 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module GEMDAQStatusSource/'GEMDAQStatusSource'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: MuonDigiCollection<unsigned short,GEMAMC13Status>
Looking for module label: muonGEMDigis
Looking for productInstanceName: AMC13Status

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

312.0

----- Begin Fatal Exception 17-Jun-2022 14:48:14 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmoffline_step'
   [2] Calling method for module GEMDAQStatusSource/'GEMDAQStatusSource'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: MuonDigiCollection<unsigned short,GEMAMC13Status>
Looking for module label: muonGEMDigis
Looking for productInstanceName: AMC13Status

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

136.897136.897_RunCosmics2021CRUZET+RunCosmics2021CRUZET+RECOCOSDRUN3+ALCACOSDRUN3+HARVESTDCR3/step2_RunCosmics2021CRUZET+RunCosmics2021CRUZET+RECOCOSDRUN3+ALCACOSDRUN3+HARVESTDCR3.log
136.899136.899_RunCosmics2021CRAFT+RunCosmics2021CRAFT+RECOCOSDRUN3+ALCACOSDRUN3+HARVESTDCR3/step2_RunCosmics2021CRAFT+RunCosmics2021CRAFT+RECOCOSDRUN3+ALCACOSDRUN3+HARVESTDCR3.log
138.1138.1_PromptCosmics+RunCosmics2021+RECOCOSDPROMPTRUN3+ALCACOSDPROMPTRUN3+HARVESTDCPROMPTRUN3/step2_PromptCosmics+RunCosmics2021+RECOCOSDPROMPTRUN3+ALCACOSDPROMPTRUN3+HARVESTDCPROMPTRUN3.log

Expand to see more relval errors ...

quark2 · 2022-06-21T17:47:05Z

Hi @jpata,

Nope, but I expect that the change would not significantly affect the runtime performance since it just adds small items that keep simple bitwise statuses, and these are not used anywhere except onlineDQM and offlineDQM.

emanueleusai · 2022-06-21T22:56:25Z

@jshlee @quark2 we need a 12_3 backport for DQM online

jpata · 2022-06-22T11:27:48Z

enable profiling

jpata · 2022-06-22T11:27:54Z

@cmsbuild please test

cmsbuild · 2022-06-22T19:41:12Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b4b802/25693/summary.html
COMMIT: 06de030
CMSSW: CMSSW_12_5_X_2022-06-22-1100/el8_amd64_gcc10
Additional Tests: PROFILING
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38398/25693/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 60 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3658917
DQMHistoTests: Total failures: 21
DQMHistoTests: Total nulls: 64
DQMHistoTests: Total successes: 3658810
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 87826.14899999998 KiB( 49 files compared)
DQMHistoSizes: changed ( 10024.0,... ): 200.027 KiB GEM/RecHits
DQMHistoSizes: changed ( 10024.0,... ): 7.764 KiB GEM/Digis
DQMHistoSizes: changed ( 11634.0,... ): 2034.563 KiB GEM/EventInfo
DQMHistoSizes: changed ( 11634.0,... ): 456.305 KiB GEM/RecHits
DQMHistoSizes: changed ( 11634.0,... ): 202.308 KiB GEM/DAQStatus
DQMHistoSizes: changed ( 11634.0,... ): 77.393 KiB GEM/Digis
DQMHistoSizes: changed ( 138.4,... ): 1798.537 KiB GEM/EventInfo
DQMHistoSizes: changed ( 138.4,... ): 353.492 KiB GEM/RecHits
DQMHistoSizes: changed ( 138.4,... ): 194.687 KiB GEM/DAQStatus
DQMHistoSizes: changed ( 138.4,... ): 15.527 KiB GEM/Digis
DQMHistoSizes: changed ( 23234.0 ): ...
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

jpata · 2022-06-23T06:35:30Z

+reconstruction

from the reco side, it's just setting keepDAQStatus=true in the GEM EventFilter, which keeps some additional information for DQM
no runtime impact seen
no reco changes

cmsbuild · 2022-06-23T06:35:57Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2022-06-23T12:00:27Z

All phase2 workflows run in the automatic tests show now the following warning message in the step3 logs:

%MSG-w GEMDAQStatusSource:  GEMDAQStatusSource:GEMDAQStatusSource  22-Jun-2022 14:21:02 CEST Run: 1 Event: 1
DAQ sources from muonGEMDigis are not found

@quark2 could you please check?

quark2 · 2022-06-23T13:14:59Z

Hi @perrotta,

This is expected in the relval stage. In this stage, some components used in the new version of offlineDQM (more specifically, GEM electronics status) are not produced. The plots related to the components are not drawn in the relval setup, so there is no problem with the result.

perrotta · 2022-06-23T13:35:54Z

Hi @perrotta,

This is expected in the relval stage. In thnis stage, some components used in the new version of offlineDQM (more specifically, GEM electronics status) are not produced. The plots related to the components are not drawn in the relval setup, so there is no problem with the result.

Sorry, I can't understand why do you say that it only depends on the RelVal.
What is expected to be different in MC productions or real data taking ?

quark2 · 2022-06-23T14:32:03Z

The electronics status of GEM is not kept in RelVal, while it is saved in real data taking and MC production. There is no reason for the keeping, as I know. It's the difference, which causes the warning because plots drawn in GEMDAQStatusSource need the electronics status. But for the same reason of the absence, we don't need to draw the plots in RelVal, so the warning message is not harmful at all.

perrotta · 2022-06-24T10:30:50Z

+1

davidlange6 · 2022-10-11T07:08:29Z

hum, no clue why this message sent months ago appeared..

davidlange6 · 2022-10-11T08:32:29Z

Unless policies have changed, merging this is not required for online dqm use.

…

On Jun 20, 2022, at 5:17 PM, Tamas Vami ***@***.***> wrote: Hi @jshlee my jurisdiction is to packages that touch the alca signature, I can make the PR urgent if that helps, but not too much else — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you are subscribed to this thread.

More on GE21

a262ae5

cmsbuild added this to the CMSSW_12_5_X milestone Jun 16, 2022

cmsbuild added code-checks-pending dqm-pending orp-pending pending-signatures tests-pending labels Jun 16, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 16, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 16, 2022

cmsbuild added tests-rejected and removed tests-started labels Jun 16, 2022

More configuration on OfflineDQM

aa3a8f5

cmsbuild added code-checks-pending tests-pending and removed tests-rejected code-checks-approved labels Jun 17, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 17, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 17, 2022

cmsbuild added tests-started and removed tests-approved labels Jun 22, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 22, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Jun 23, 2022

cmsbuild added orp-approved and removed orp-pending labels Jun 24, 2022

cmsbuild merged commit d89c48f into cms-sw:master Jun 24, 2022

This was referenced Jun 24, 2022

More plots from GE21 for the GEM DQMs, a backport to 12_3_X #38499

Closed

More plots from GE21 for the GEM DQMs, a backport to 12_4_X #38500

Merged

cmsbuild mentioned this pull request Jun 24, 2022

Adding Xi, Chi_c and Chi_b to the matrix for RelVals. #38494

Merged

This was referenced Jun 27, 2022

Better configurations for the DAQ status plots of GEM onlineDQM #38526

Merged

Updates on GUI of GEM onlineDQM and offlineDQM dmwm/deployment#1160

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More plots from GE21 for the GEM DQMs #38398

More plots from GE21 for the GEM DQMs #38398

quark2 commented Jun 16, 2022

cmsbuild commented Jun 16, 2022

cmsbuild commented Jun 16, 2022

jshlee commented Jun 16, 2022

cmsbuild commented Jun 16, 2022

cmsbuild commented Jun 17, 2022

cmsbuild commented Jun 17, 2022

jshlee commented Jun 17, 2022

cmsbuild commented Jun 17, 2022

quark2 commented Jun 21, 2022

emanueleusai commented Jun 21, 2022

jpata commented Jun 22, 2022

jpata commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

jpata commented Jun 23, 2022

cmsbuild commented Jun 23, 2022

perrotta commented Jun 23, 2022

quark2 commented Jun 23, 2022

perrotta commented Jun 23, 2022

quark2 commented Jun 23, 2022

perrotta commented Jun 24, 2022

davidlange6 commented Oct 11, 2022

davidlange6 commented Oct 11, 2022 via email

More plots from GE21 for the GEM DQMs #38398

More plots from GE21 for the GEM DQMs #38398

Conversation

quark2 commented Jun 16, 2022

PR description:

PR validation:

cmsbuild commented Jun 16, 2022

cmsbuild commented Jun 16, 2022

jshlee commented Jun 16, 2022

cmsbuild commented Jun 16, 2022

Unit Tests

RelVals

RelVals-INPUT

cmsbuild commented Jun 17, 2022

cmsbuild commented Jun 17, 2022

jshlee commented Jun 17, 2022

cmsbuild commented Jun 17, 2022

RelVals

RelVals-INPUT

quark2 commented Jun 21, 2022

emanueleusai commented Jun 21, 2022

jpata commented Jun 22, 2022

jpata commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

Comparison Summary

jpata commented Jun 23, 2022

cmsbuild commented Jun 23, 2022

perrotta commented Jun 23, 2022

quark2 commented Jun 23, 2022

perrotta commented Jun 23, 2022

quark2 commented Jun 23, 2022

perrotta commented Jun 24, 2022

davidlange6 commented Oct 11, 2022

davidlange6 commented Oct 11, 2022 via email