DT DQM: attempt to clean Legacy (Run1) DDU settings #27996

jfernan2 · 2019-09-16T08:19:20Z

PR description:

This PR tries to clean DT DQM code from DDU settings belonging to Legacy (Run1). Also an empty plot in TM TriggerEfficiency is fixed by using an appropriate collection of digis

PR validation:

Tested with runTheMatrix.py -l limited -i all --ibeos

if this PR is a backport please specify the original PR:

No need to backport. Cleaning targeting Run3

jfernan2 · 2019-09-16T08:19:29Z

please test

cmsbuild · 2019-09-16T08:19:48Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-09-16T08:28:29Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27996/11899

This PR adds an extra 64KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27996/11899/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27996/11899/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2019-09-16T08:31:03Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-09-16T08:36:59Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-27996/11900

This PR adds an extra 68KB to repository

cmsbuild · 2019-09-16T08:37:28Z

A new Pull Request was created by @jfernan2 for master.

It involves the following packages:

DQM/DTMonitorClient
DQM/DTMonitorModule

@andrius-k, @kmaeshima, @schneiml, @cmsbuild, @jfernan2, @fioriNTU can you please review it and eventually sign? Thanks.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

jfernan2 · 2019-09-16T08:37:46Z

please test

cmsbuild · 2019-09-16T08:37:56Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2522/console Started: 2019/09/16 10:38

cmsbuild · 2019-09-16T10:58:50Z

-1

Tested at: 76844c0

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-76817d/2522/summary.html

I found follow errors while testing this PR

Failed tests: RelVals

RelVals:

When I ran the RelVals I found an error in the following workflows:
1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

10042.0 step3

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

10024.0 step3

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017/step3_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+HARVESTFull_2017+ALCAFull_2017+NanoFull_2017.log

cmsbuild · 2019-09-16T10:58:53Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2019-09-27T16:39:31Z

+1
Tested at: 09220ab
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-76817d/2705/summary.html

cmsbuild · 2019-09-27T16:39:35Z

Comparison job queued.

cmsbuild · 2019-09-27T18:46:25Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-76817d/2705/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2956833
DQMHistoTests: Total failures: 79
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2956413
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -682.911 KiB( 33 files compared)
DQMHistoSizes: changed ( 4.53,... ): -12.478 KiB DT/04-LocalTrigger-DDU
DQMHistoSizes: changed ( 140.53 ): -5.616 KiB DT/04-LocalTrigger-DDU
DQMHistoSizes: changed ( 4.22,... ): -297.783 KiB AlCaReco/DtCalibSynch
DQMHistoSizes: changed ( 136.731 ): -6.861 KiB DT/04-LocalTrigger-DDU
Checked 145 log files, 15 edm output root files, 34 DQM output files

jfernan2 · 2019-09-28T14:55:52Z

+1
Changes expected due to the cleaning

cmsbuild · 2019-09-28T14:56:21Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-10-01T20:39:46Z

+1

Martin-Grunewald · 2019-10-02T07:11:58Z

Looks like this PR creates a problem in a L1T stage-1 TSG test workflow:

----- Begin Fatal Exception 02-Oct-2019 07:58:11 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 256677 lumi: 380 event: 397024298 stream: 2
   [1] Calling method for module DTTriggerEfficiencyTask/'dtTriggerEfficiencyMonitor'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: L1MuDTChambPhContainer
Looking for module label: twinMuxStage2Digis
Looking for productInstanceName: PhIn

----- End Fatal Exception -------------------------------------------------

(no stage2 products in a stage1 workflow):

# with command line options: RelVal --step=RAW2DIGI,L1Reco,RECO,EI,PAT,VALIDATION,DQM --conditions=auto:run2_mc_Fake1 --filein=file:RelVal_DigiL1RawHLT_Fake1_MC.root --custom_conditions= --fileout=RelVal_RECO_Fake1_MC.root --number=100 --mc --no_exec --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO --eventcontent=RECOSIM,MINIAODSIM,DQM --customise=HLTrigger/Configuration/CustomConfigs.Base --era=Run2_25ns --customise= --scenario=pp --python_filename=RelVal_RECO_Fake1_MC.py --processName=RECO --runUnscheduled

jfernan2 · 2019-10-02T07:23:02Z

@Martin-Grunewald can you specify which wf? Is it a standard one? Run1 wf should not read that collection if Era treatment is applied, that's what prevents this line:
https://github.com/cms-sw/cmssw/pull/27996/files#diff-75afa721b61352b2086ab193486f0a68R28

Looks like this PR creates a problem in a Run-1 TSG test workflow:

----- Begin Fatal Exception 02-Oct-2019 07:58:11 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 256677 lumi: 380 event: 397024298 stream: 2
   [1] Calling method for module DTTriggerEfficiencyTask/'dtTriggerEfficiencyMonitor'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: L1MuDTChambPhContainer
Looking for module label: twinMuxStage2Digis
Looking for productInstanceName: PhIn

----- End Fatal Exception -------------------------------------------------

Martin-Grunewald · 2019-10-02T07:24:54Z

DQM/DTMonitorModule/python/dtTriggerEfficiencyTask_cfi.py

@@ -30,6 +25,5 @@
 # Modify for running in run 2 2016 data
 #
 from Configuration.Eras.Modifier_run2_common_cff import run2_common
-run2_common.toModify( dtTriggerEfficiencyMonitor, checkRPCtriggers = cms.untracked.bool(False),processDDU = cms.untracked.bool(False))
-
+run2_common.toModify( dtTriggerEfficiencyMonitor, inputTagTM = cms.untracked.InputTag('twinMuxStage2Digis:PhIn'))


Should likely not be done for an L1T stage-1 workflow...

Can you suggest an Era or equivalent modifier then? And a wf number to test it?
Thanks

Hmm, perhaps stage2L1Trigger (so that it does not get applied to stage1L1Trigger)

Martin-Grunewald · 2019-10-02T07:26:13Z

It is a Run-2 workflow but with L1T stage-1 setup (before L1T went to stage-2).
(I corrected the comment).
It is a test workflow used by TSG.

jfernan2 · 2019-10-02T07:28:10Z

It is a Run-2 workflow but with L1T stage-1 setup (before L1T went to stage-2).

2016 onwards excluding 2015 perhaps?

Martin-Grunewald · 2019-10-02T07:32:27Z

Stage-1 L1T was used before stage-2 L1T, in 2015/16

jfernan2 · 2019-10-02T07:35:13Z

Then I believe
from Configuration.Eras.Modifier_run2_25ns_specific_cff import run2_25ns_specific
run2_25ns_specific.toModify(
is the right one.
I will create a fix. Can you tell me a wf number to test it?
Apologies for the inconvenience

Martin-Grunewald · 2019-10-02T07:36:01Z

Many standard workflows (IB relvals) now fail with this, see:
https://cms-sw.github.io/relvalLogDetail.html#slc7_amd64_gcc820;CMSSW_11_0_X_2019-10-01-2300

Martin-Grunewald · 2019-10-02T07:45:08Z

Wouldn't stage2L1Trigger be more targeted?

jfernan2 · 2019-10-02T07:47:29Z

OK, I will try that then.

jfernan2 · 2019-10-02T10:15:27Z

Fixing this problem in: PR #28105
Thanks @Martin-Grunewald for spotting

jfernan2 added 2 commits September 16, 2019 10:12

DDU cleanup

6cd3383

Fix TM Trigger Efficiency plot

407d0a3

cmsbuild added this to the CMSSW_11_0_X milestone Sep 16, 2019

cmsbuild added code-checks-pending comparison-pending dqm-pending orp-pending pending-signatures tests-pending labels Sep 16, 2019

cmsbuild added code-checks-rejected and removed code-checks-pending labels Sep 16, 2019

Code format changes

76844c0

cmsbuild added code-checks-pending and removed code-checks-rejected labels Sep 16, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 16, 2019

cmsbuild added tests-started and removed tests-pending labels Sep 16, 2019

cmsbuild removed comparison-pending tests-started labels Sep 16, 2019

cmsbuild added tests-approved and removed tests-started labels Sep 27, 2019

cmsbuild added comparison-available and removed comparison-pending labels Sep 27, 2019

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Sep 28, 2019

cmsbuild mentioned this pull request Oct 1, 2019

DQM: Restructure the edm::one DQMEDAnalyzer base classes #28099

Merged

cmsbuild added orp-approved and removed orp-pending labels Oct 1, 2019

cmsbuild merged commit f795bce into cms-sw:master Oct 1, 2019

Martin-Grunewald reviewed Oct 2, 2019

View reviewed changes

jfernan2 mentioned this pull request Oct 2, 2019

DT DQM: Fixing inconsistency in L1Tstage2 #28105

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DT DQM: attempt to clean Legacy (Run1) DDU settings #27996

DT DQM: attempt to clean Legacy (Run1) DDU settings #27996

jfernan2 commented Sep 16, 2019 •

edited

jfernan2 commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

jfernan2 commented Sep 16, 2019

cmsbuild commented Sep 16, 2019 •

edited

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 27, 2019

cmsbuild commented Sep 27, 2019

cmsbuild commented Sep 27, 2019

jfernan2 commented Sep 28, 2019

cmsbuild commented Sep 28, 2019

fabiocos commented Oct 1, 2019

Martin-Grunewald commented Oct 2, 2019 •

edited

jfernan2 commented Oct 2, 2019

Martin-Grunewald Oct 2, 2019

jfernan2 Oct 2, 2019

Martin-Grunewald Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019 •

edited

jfernan2 commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

DT DQM: attempt to clean Legacy (Run1) DDU settings #27996

DT DQM: attempt to clean Legacy (Run1) DDU settings #27996

Conversation

jfernan2 commented Sep 16, 2019 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR:

jfernan2 commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

jfernan2 commented Sep 16, 2019

cmsbuild commented Sep 16, 2019 • edited

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 16, 2019

cmsbuild commented Sep 27, 2019

cmsbuild commented Sep 27, 2019

cmsbuild commented Sep 27, 2019

jfernan2 commented Sep 28, 2019

cmsbuild commented Sep 28, 2019

fabiocos commented Oct 1, 2019

Martin-Grunewald commented Oct 2, 2019 • edited

jfernan2 commented Oct 2, 2019

Martin-Grunewald Oct 2, 2019

Choose a reason for hiding this comment

jfernan2 Oct 2, 2019

Choose a reason for hiding this comment

Martin-Grunewald Oct 2, 2019

Choose a reason for hiding this comment

Martin-Grunewald commented Oct 2, 2019 • edited

jfernan2 commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

Martin-Grunewald commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

jfernan2 commented Oct 2, 2019

jfernan2 commented Sep 16, 2019 •

edited

cmsbuild commented Sep 16, 2019 •

edited

Martin-Grunewald commented Oct 2, 2019 •

edited

Martin-Grunewald commented Oct 2, 2019 •

edited