Phase2 tracker DQM package #28432

gsaha009 · 2019-11-20T10:28:47Z

PR description:

In this PR, two sub packages are created for Phase2 tracker DQM and Validation.

Creating DQM/SiTrackerPhase2 and Validation/SiTrackerPhase2V packages.
The main source code already existed in SimTracker/SiPhase2Digitizer package and was used in a standalone mode.
Changes in DQM step: moved Phase2TrackerMonitorDigi module from SimTracker/SiPhase2Digitizer/test to DQM/SiTrackerPhase2/plugins. Updated DQM offline configs and autoDQM config to include this module in standard phase2 workflow.
Changes in Validation step: moved Phase2TrackerValidateDigi module from SimTracker/SiPhase2Digitizer/test to Validation/SiTrackerPhase2V/plugins. Updated Validation configs and autoValidation config to include this module in standard phase2 workflow.
Config files for 3, 4 are named following central DQM recommendations (twiki)

PR validation:

Tests done with workflows for phase2 : 20034.0, 21234.0, 22834.0, 23234.0

if this PR is a backport please specify the original PR: -NA

…ase2 workflows

…ase2 workflows --Edited

cmsbuild · 2019-11-20T10:29:19Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-20T10:35:10Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12831

This PR adds an extra 44KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/autoDQM.py modified in PR(s): Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12831/code-checks.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12831/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly
code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12831/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12831/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2019-11-20T11:33:54Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-20T11:42:49Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12837

This PR adds an extra 44KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/autoDQM.py modified in PR(s): Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12837/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12837/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2019-11-20T12:46:53Z

The code-checks are being triggered in jenkins.

cmsbuild · 2019-11-20T12:51:28Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28432/12839

This PR adds an extra 52KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File DQMOffline/Configuration/python/DQMOffline_SecondStep_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/DQMOffline_cff.py modified in PR(s): Squash all Patatrack developments on top of CMSSW_11_3_0_pre5 #27983
- File DQMOffline/Configuration/python/autoDQM.py modified in PR(s): Fix DQM ZB workflows and moved DQM FS configuration to DQM package #28351

cmsbuild · 2019-11-25T12:28:51Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3610/console Started: 2019/11/25 13:29

mmusich · 2019-11-25T12:34:03Z

@fabiocos

the new packages basically contain a single plugin each, do you expect a further expansion?

much. This is the first stub of the still undeveloped phase2 tracker DQM.

cmsbuild · 2019-11-25T14:04:02Z

+1
Tested at: 2fef181
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9db8bc/3610/summary.html

cmsbuild · 2019-11-25T14:04:06Z

Comparison job queued.

cmsbuild · 2019-11-25T15:23:03Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9db8bc/3610/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-9db8bc/22834.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D48_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D48+RecoFullGlobal_2026D48+HARVESTFullGlobal_2026D48
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-9db8bc/23234.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2026D49_GenSimHLBeamSpotFull14+DigiFullTrigger_2026D49+RecoFullGlobal_2026D49+HARVESTFullGlobal_2026D49

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2787584
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2787242
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 628826.268 KiB( 33 files compared)
DQMHistoSizes: changed ( 20034.0,... ): 151719.350 KiB TrackerPhase2ITDigi/DigiMonitor
DQMHistoSizes: changed ( 20034.0,... ): 5431.483 KiB TrackerPhase2OTDigi/DigiMonitor
DQMHistoSizes: changed ( 20034.0,... ): 27.867 KiB TrackerPhase2OTDigi/SimTrackInfo
DQMHistoSizes: changed ( 20034.0,... ): 27.867 KiB TrackerPhase2ITDigi/SimTrackInfo
Checked 147 log files, 16 edm output root files, 34 DQM output files

fabiocos · 2019-11-25T16:49:05Z

@jfernan2 @civanch @kpedro88 the update is purely technical, I will consider your previous signatures as valid. Please keep in mind that this update will not be innocent in terms of performances

jfernan2 · 2019-11-25T16:50:31Z

+1

kpedro88 · 2019-11-25T16:54:19Z

+upgrade

fabiocos · 2019-11-25T21:10:35Z

+1

fabiocos · 2019-11-25T21:11:21Z

merge

fabiocos · 2019-11-26T22:00:26Z

@gsaha009 @mmusich I see there is a problem in the IB in test wf 20634.99:

----- Begin Fatal Exception 26-Nov-2019 18:28:11 CET-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 2 event: 105 stream: 3
   [1] Calling method for module Phase2TrackerMonitorDigi/'pixDigiMon'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: edm::DetSetVector&lt;PixelDigi&gt;
Looking for module label: simSiPixelDigis
Looking for productInstanceName: Pixel

----- End Fatal Exception -------------------------------------------------

mmusich · 2019-11-26T23:08:19Z

@fabiocos as far as I see it's the only affected wf (a bit peculiar). Is it the only premixed wf in the upgrade matrix?

mmusich · 2019-11-26T23:17:56Z

DQM/SiTrackerPhase2/python/Phase2TrackerMonitorDigi_cfi.py

    ),
    RZPositionMapH = cms.PSet(
        Nxbins = cms.int32(3000),
        xmin   = cms.double(-3000.),
        xmax   = cms.double(3000.),
        Nybins = cms.int32(1250),
        ymin   = cms.double(0.),
-        ymax   = cms.double(1250.)
+        ymax   = cms.double(1250.),
+        switch = cms.bool(False)
    )
 )


@gsaha009.
I think that in this file InnerPixelDigiSource should be customized in the case of premixing.
Something like:

from Configuration.ProcessModifiers.premix_stage2_cff import premix_stage2 (premix_stage2 & phase2_tracker).toModify(digiMon, InnerPixelDigiSource = "mixData:Pixel" )

or something similar (see e.g. https://cmssdt.cern.ch/dxr/CMSSW/source/RecoLocalTracker/SiPixelClusterizer/python/SiPixelClusterizer_cfi.py#31)

mmusich · 2019-11-27T04:59:51Z

@fabiocos issue is addressed here: https://github.com/cms-sw/cmssw/pull/28486/files

gsaha009 added 4 commits November 20, 2019 01:38

Removing files being moved to the new package

6b880b8

Create new package for TrackerPhase2 DQM and Validation

8cfb700

Updating core DQM/Validation configs to include the new modules in Ph…

f77e2ff

…ase2 workflows

Updating core DQM/Validation configs to include the new modules in Ph…

0d41262

…ase2 workflows --Edited

cmsbuild added this to the CMSSW_11_0_X milestone Nov 20, 2019

cmsbuild added code-checks-pending comparison-pending dqm-pending new-package-pending orp-pending pending-signatures simulation-pending tests-pending upgrade-pending labels Nov 20, 2019

cmsbuild added code-checks-rejected and removed code-checks-pending labels Nov 20, 2019

updates to address code checks

27d14d9

cmsbuild added code-checks-pending and removed code-checks-rejected labels Nov 20, 2019

cmsbuild added code-checks-rejected and removed code-checks-pending labels Nov 20, 2019

updates to address code format checks

4d3c1e8

cmsbuild added code-checks-pending and removed code-checks-rejected labels Nov 20, 2019

cmsbuild added tests-started and removed tests-pending labels Nov 25, 2019

cmsbuild added tests-approved and removed tests-started labels Nov 25, 2019

cmsbuild added comparison-available and removed comparison-pending labels Nov 25, 2019

cmsbuild added dqm-approved and removed dqm-pending labels Nov 25, 2019

cmsbuild added upgrade-approved and removed upgrade-pending labels Nov 25, 2019

cmsbuild added orp-approved and removed orp-pending labels Nov 25, 2019

cmsbuild merged commit b297706 into cms-sw:master Nov 25, 2019

cmsbuild removed the new-package-pending label Nov 26, 2019

mmusich reviewed Nov 26, 2019

View reviewed changes

mmusich mentioned this pull request Nov 27, 2019

Phase2 Tracker DQM: fix missing modifiers #28486

Merged

kpedro88 mentioned this pull request Mar 3, 2020

Inclusion of Pixel 3D technology in the innermost layers of Phase 2 Inner Tracker #29018

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase2 tracker DQM package #28432

Phase2 tracker DQM package #28432

gsaha009 commented Nov 20, 2019 •

edited

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 25, 2019 •

edited

mmusich commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

fabiocos commented Nov 25, 2019

jfernan2 commented Nov 25, 2019

kpedro88 commented Nov 25, 2019

fabiocos commented Nov 25, 2019

fabiocos commented Nov 25, 2019

fabiocos commented Nov 26, 2019 •

edited

mmusich commented Nov 26, 2019

mmusich Nov 26, 2019

mmusich commented Nov 27, 2019

Phase2 tracker DQM package #28432

Phase2 tracker DQM package #28432

Conversation

gsaha009 commented Nov 20, 2019 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR: -NA

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 20, 2019

cmsbuild commented Nov 25, 2019 • edited

mmusich commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

cmsbuild commented Nov 25, 2019

fabiocos commented Nov 25, 2019

jfernan2 commented Nov 25, 2019

kpedro88 commented Nov 25, 2019

fabiocos commented Nov 25, 2019

fabiocos commented Nov 25, 2019

fabiocos commented Nov 26, 2019 • edited

mmusich commented Nov 26, 2019

mmusich Nov 26, 2019

Choose a reason for hiding this comment

mmusich commented Nov 27, 2019

gsaha009 commented Nov 20, 2019 •

edited

cmsbuild commented Nov 25, 2019 •

edited

fabiocos commented Nov 26, 2019 •

edited