DQM: Remove DQMDcsInfo #28829

schneiml · 2020-01-30T11:56:35Z

PR description:

The Info/EventInfo/reportSummaryMap DCS plot was broken since a long time. I have never seen the 'Physics Declared' actually trigger, and since some changes early 2019 also the actual DCS bit information is completely broken (I have not that much confidence that it was actually correct before, e.g. when some lumisections were not processed).

DQMProvInfo provides roughly the same functionality (and more), is much
simpler and actually works, and is used in online since a long time.

This PR replaces DQMDcsInfo with DQMProvInfo also in offline.

PR validation:

Runs 136.85, more to be seen. DQMGUI render plugins need to be checked if they work on the new plot also in offline.

Update: runs and seems to work correctly, Render Plugin handles both plots correctly.

if this PR is a backport please specify the original PR and why you need to backport that PR:

We might want to backport this in the future to fix the DCS plot for the coming UL rereco.

cmsbuild · 2020-01-30T11:56:59Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-01-30T12:02:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28829/13553

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMServices/Components/plugins/DQMDcsInfo.cc modified in PR(s): DQM: Switch to edm::stream for DQMEDAnalyzer #28813, DQM: new DQMStore. #28622
- File DQMServices/Components/plugins/SealModule.cc modified in PR(s): DQM: Switch to edm::stream for DQMEDAnalyzer #28813, DQM: new DQMStore. #28622, DQM: New EventInfo module for Harvesting #28821

cmsbuild · 2020-01-30T12:03:21Z

A new Pull Request was created by @schneiml (Marcel Schneider) for master.

It involves the following packages:

DQMOffline/Configuration
DQMOffline/L1Trigger
DQMOffline/Muon
DQMServices/Components

@benkrikler, @kmaeshima, @schneiml, @andrius-k, @cmsbuild, @rekovic, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@barvic, @kreczko, @battibass, @threus, @abbiendi, @jhgoh, @calderona, @cericeci, @HuguesBrun, @Fedespring, @ptcox, @bellan, @trocino, @folguera, @rociovilar this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

schneiml · 2020-01-30T13:00:44Z

please test

some new plots expected, as DQMProvInfo records more things compared to DQMDcsInfo. Also, ideally I caught all the references to DQMDcsInfo and nothing crashes.

cmsbuild · 2020-01-30T13:01:07Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4429/console Started: 2020/01/30 14:44

cmsbuild · 2020-01-30T14:48:59Z

-1

Tested at: 85a7032

CMSSW: CMSSW_11_1_X_2020-01-30-1100
SCRAM_ARCH: slc7_amd64_gcc820
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d45547/4429/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
8.0 step3

runTheMatrix-results/8.0_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS/step3_BeamHalo+BeamHalo+DIGICOS+RECOCOS+ALCABH+HARVESTCOS.log

7.3 step3

runTheMatrix-results/7.3_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18/step3_CosmicsSPLoose_UP18+CosmicsSPLoose_UP18+DIGICOS_UP18+RECOCOS_UP18+ALCACOS_UP18+HARVESTCOS_UP18.log

9.0 step3

runTheMatrix-results/9.0_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST/step3_Higgs200ChargedTaus+Higgs200ChargedTaus+DIGI+RECO+HARVEST.log

25.0 step3

runTheMatrix-results/25.0_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT/step3_TTbar+TTbar+DIGI+RECOAlCaCalo+HARVEST+ALCATT.log

1306.0 step3

runTheMatrix-results/1306.0_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15/step3_SingleMuPt1_UP15+SingleMuPt1_UP15+DIGIUP15+RECOUP15+HARVESTUP15.log

1330.0 step3

runTheMatrix-results/1330.0_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15/step3_ZMM_13+ZMM_13+DIGIUP15+RECOUP15_L1TMuDQM+HARVESTUP15_L1TMuDQM+NANOUP15.log

158.0 step3

runTheMatrix-results/158.0_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO/step3_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO.log

10042.0 step3

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

25202.0 step3

runTheMatrix-results/25202.0_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25/step3_TTbar_13+TTbar_13+DIGIUP15_PU25+RECOUP15_PU25+HARVESTUP15_PU25+NANOUP15_PU25.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

10824.0 step3

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+HARVESTFull_2018+ALCAFull_2018+NanoFull_2018.log

11634.0 step3

runTheMatrix-results/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021.log

10224.0 step3

runTheMatrix-results/10224.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU+NanoFull_2017PU.log

12434.0 step3

runTheMatrix-results/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023.log

20034.0 step3

runTheMatrix-results/20034.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D35_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D35+RecoFullGlobal_2026D35+HARVESTFullGlobal_2026D35.log

20434.0 step3

runTheMatrix-results/20434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D41+RecoFullGlobal_2026D41+HARVESTFullGlobal_2026D41/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D41_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D41+RecoFullGlobal_2026D41+HARVESTFullGlobal_2026D41.log

23234.0 step3

runTheMatrix-results/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49.log

21234.0 step3

runTheMatrix-results/21234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D44_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D44+RecoFullGlobal_2026D44+HARVESTFullGlobal_2026D44/step3_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D44_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D44+RecoFullGlobal_2026D44+HARVESTFullGlobal_2026D44.log

250202.181 step4

runTheMatrix-results/250202.181_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25/step4_TTbar_13UP18+TTbar_13UP18+PREMIXUP18_PU25+DIGIPRMXLOCALUP18_PU25+RECOPRMXUP18_PU25+HARVESTUP18_PU25.log

cmsbuild · 2020-01-30T14:49:03Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

schneiml · 2020-01-31T16:28:06Z

It seems we'll need to have a look at optional vs. required dependencies on the old vs. new DCS products.

schneiml · 2020-02-04T11:32:46Z

So, actually what crashed here are all MC based workflows. While the old DQMDcsInfo silently ignores the absence of a DCS record, DQMProvInfo does not...

cmsbuild · 2020-02-11T13:57:15Z

Pull request #28829 was updated. @benkrikler, @kmaeshima, @schneiml, @andrius-k, @cmsbuild, @rekovic, @jfernan2, @fioriNTU can you please check and sign again.

schneiml · 2020-02-11T14:02:15Z

please test

Rebased to remove the merge conflict. The new behaviour was considered fine so far, so we'll move forward here soon.

@boudoul The previous test result produced a nice comparison: https://tinyurl.com/tsq8649
Note that there is only a few events from a single lumi processed, which the new (left) plot correctly shows while the old (right) plot shows... something. The new plot also adds the beam status, compared to the old one.

cmsbuild · 2020-02-11T14:02:42Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4592/console Started: 2020/02/11 15:06

cmsbuild · 2020-02-11T15:21:11Z

+1
Tested at: 6f11346
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d45547/4592/summary.html
CMSSW: CMSSW_11_1_X_2020-02-10-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-02-11T15:21:14Z

Comparison job queued.

cmsbuild · 2020-02-11T16:57:58Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-d45547/4592/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 3 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2693951
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 27
DQMHistoTests: Total successes: 2693603
DQMHistoTests: Total skipped: 319
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 22674.563 KiB( 33 files compared)
DQMHistoSizes: changed ( 1000.0 ): 719.670 KiB Info/EventInfo
DQMHistoSizes: changed ( 1000.0,... ): 117.332 KiB Info/LhcInfo
DQMHistoSizes: changed ( 1000.0,... ): 0.228 KiB Info/ProvInfo
DQMHistoSizes: changed ( 1000.0,... ): -0.125 KiB Info/DcsInfo
DQMHistoSizes: changed ( 1000.0,... ): -0.083 KiB Info/CMSSWInfo
DQMHistoSizes: changed ( 10024.0,... ): 725.498 KiB Info/EventInfo
DQMHistoSizes: changed ( 136.731 ): 724.154 KiB Info/EventInfo
DQMHistoSizes: changed ( 136.793 ): 725.467 KiB Info/EventInfo
DQMHistoSizes: changed ( 136.874 ): 718.139 KiB Info/EventInfo
DQMHistoSizes: changed ( 140.53 ): 722.076 KiB Info/EventInfo
DQMHistoSizes: changed ( 140.56 ): ...
Checked 147 log files, 16 edm output root files, 34 DQM output files

schneiml · 2020-02-12T08:59:10Z

+1

Seems fine on our side. Be prepared for some false positives in the bin-to-bin comparison once this is merged: We'll need to put some of the ProvInfo values into the blacklist (if they are not there already).

rekovic · 2020-02-17T08:57:43Z

+1

cmsbuild · 2020-02-17T08:58:08Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2020-02-17T18:19:16Z

+1

cmsbuild added this to the CMSSW_11_1_X milestone Jan 30, 2020

cmsbuild added code-checks-pending comparison-pending dqm-pending l1-pending orp-pending pending-signatures tests-pending labels Jan 30, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jan 30, 2020

cmsbuild added tests-started and removed tests-pending labels Jan 30, 2020

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Jan 30, 2020

This was referenced Jan 30, 2020

DQM: New EventInfo module for Harvesting #28821

Merged

DQM: Switch to edm::stream for DQMEDAnalyzer #28813

Merged

cmsbuild mentioned this pull request Feb 3, 2020

DQM: new DQMStore. #28622

Merged

cmsbuild removed code-checks-approved comparison-notrun labels Feb 4, 2020

cmsbuild added the code-checks-approved label Feb 11, 2020

cmsbuild added tests-started and removed tests-pending labels Feb 11, 2020

cmsbuild added tests-approved and removed tests-started labels Feb 11, 2020

cmsbuild added comparison-available and removed comparison-pending labels Feb 11, 2020

cmsbuild added dqm-approved and removed dqm-pending labels Feb 12, 2020

cmsbuild added fully-signed l1-approved and removed l1-pending pending-signatures labels Feb 17, 2020

cmsbuild added orp-approved and removed orp-pending labels Feb 17, 2020

cmsbuild merged commit 6df4233 into cms-sw:master Feb 17, 2020

cmsbuild mentioned this pull request Feb 18, 2020

customization files needed for CSC reco objects study #28979

Merged

This was referenced Mar 10, 2020

DQM: Fix dcsRecord input tag in offline for DQMProvInfo #29161

Merged

DQM: Blacklist some ProvInfo plots #29177

Merged

schneiml mentioned this pull request Mar 30, 2020

DQM: fix some autoDQM.py harvesting sequences #29343

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DQM: Remove DQMDcsInfo #28829

DQM: Remove DQMDcsInfo #28829

schneiml commented Jan 30, 2020 •

edited

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

schneiml commented Jan 30, 2020

cmsbuild commented Jan 30, 2020 •

edited

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

schneiml commented Jan 31, 2020

schneiml commented Feb 4, 2020

cmsbuild commented Feb 11, 2020

schneiml commented Feb 11, 2020

cmsbuild commented Feb 11, 2020 •

edited

cmsbuild commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

schneiml commented Feb 12, 2020

rekovic commented Feb 17, 2020

cmsbuild commented Feb 17, 2020

silviodonato commented Feb 17, 2020

DQM: Remove DQMDcsInfo #28829

DQM: Remove DQMDcsInfo #28829

Conversation

schneiml commented Jan 30, 2020 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

schneiml commented Jan 30, 2020

cmsbuild commented Jan 30, 2020 • edited

cmsbuild commented Jan 30, 2020

cmsbuild commented Jan 30, 2020

schneiml commented Jan 31, 2020

schneiml commented Feb 4, 2020

cmsbuild commented Feb 11, 2020

schneiml commented Feb 11, 2020

cmsbuild commented Feb 11, 2020 • edited

cmsbuild commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

cmsbuild commented Feb 11, 2020

schneiml commented Feb 12, 2020

rekovic commented Feb 17, 2020

cmsbuild commented Feb 17, 2020

silviodonato commented Feb 17, 2020

schneiml commented Jan 30, 2020 •

edited

cmsbuild commented Jan 30, 2020 •

edited

cmsbuild commented Feb 11, 2020 •

edited