PPS: add Run-3 reco, add geometry to GTs, update era modifier #33415

silviodonato · 2021-04-12T20:55:34Z

This is the combination of three PRs

Re-enable ctppsProtons from recoCTPPSTask in Run-3 Re-enable ctppsProtons from recoCTPPSTask in Run-3 #33266
Add PPS geometry and simulation tags to GTs Add PPS geometry and simulation tags to GTs #33404
PPS: update of era modifiers PPS: update of era modifiers #33250

More details can be found in each PR.

This PR includes also a fix in the HLT AddOn test and cmsDriver in order to be able to run on Run-2 data using --procModifier Run3. This fix affects:

HLTrigger/Configuration/test/cmsDriver.csh
HLTrigger/Configuration/python/CustomConfigs.py
Configuration/HLT/python/addOnTestsHLT.py

silviodonato · 2021-04-12T20:55:52Z

please test

cmsbuild · 2021-04-12T21:02:22Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-33415/22061

This PR adds an extra 68KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/AlCa/python/autoCond.py modified in PR(s): Add PPS geometry and simulation tags to GTs #33404
- File Configuration/Eras/python/Era_Run2_2016_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File Configuration/Eras/python/Era_Run2_2017_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File Configuration/Eras/python/Modifier_ctpps_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File Configuration/EventContent/python/EventContent_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File Configuration/StandardSequences/python/Eras.py modified in PR(s): PPS: update of era modifiers #33250, Make BadHcalMitigation enabled by default #33413
- File Configuration/StandardSequences/python/RawToDigi_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File Configuration/StandardSequences/python/Reconstruction_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File DQM/CTPPS/python/ctppsDQM_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File DQM/CTPPS/test/all_ctpps_dqm_test_from_aod_cfg.py modified in PR(s): PPS: update of era modifiers #33250
- File DQM/CTPPS/test/all_ctpps_dqm_test_from_dat_cfg.py modified in PR(s): PPS: update of era modifiers #33250
- File DQM/CTPPS/test/all_ctpps_dqm_test_from_raw_cfg.py modified in PR(s): PPS: update of era modifiers #33250
- File PhysicsTools/PatAlgos/python/slimming/miniAOD_tools.py modified in PR(s): PPS: update of era modifiers #33250
- File RecoPPS/Configuration/python/recoCTPPS_cff.py modified in PR(s): Re-enable ctppsProtons from recoCTPPSTask in Run-3 #33266
- File RecoPPS/Configuration/test/raw_data_test.py modified in PR(s): PPS: update of era modifiers #33250
- File RecoPPS/Local/plugins/CTPPSLocalTrackLiteProducer.cc modified in PR(s): PPS: update of era modifiers #33250
- File RecoPPS/Local/python/ctppsLocalTrackLiteProducer_cff.py modified in PR(s): PPS: update of era modifiers #33250
- File RecoPPS/ProtonReconstruction/python/ctppsProtons_cff.py modified in PR(s): PPS: update of era modifiers #33250

cmsbuild · 2021-04-12T21:02:49Z

A new Pull Request was created by @silviodonato (Silvio Donato) for master.

It involves the following packages:

Configuration/AlCa
Configuration/Eras
Configuration/EventContent
Configuration/StandardSequences
DQM/CTPPS
HLTrigger/Configuration
PhysicsTools/PatAlgos
RecoPPS/Configuration
RecoPPS/Local
RecoPPS/ProtonReconstruction

@andrius-k, @Martin-Grunewald, @tlampen, @pohsun, @perrotta, @yuanchao, @silviodonato, @ErnestaP, @ahmad3213, @fwyzard, @davidlange6, @jfernan2, @slava77, @jpata, @qliphy, @fabiocos, @francescobrivio, @malbouis, @kmaeshima, @christopheralanwest, @franzoni, @rvenditti can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @felicepantaleo, @forthommel, @hatakeyamak, @emilbols, @Martin-Grunewald, @mbluj, @ahinzmann, @seemasharmafnal, @mmarionncern, @makortel, @dgulhan, @jdolen, @slomeo, @ferencek, @GiacomoSguazzoni, @rovere, @VinInn, @jdamgov, @nhanvtran, @gkasieczka, @tocheng, @schoef, @mmusich, @mtosi, @fabiocos, @clelange, @jan-kaspar, @swozniewski, @JyothsnaKomaragiri, @lecriste, @gpetruc, @mariadalfonso, @andrzejnovak this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

silviodonato · 2021-04-12T21:05:22Z

please test workflow 11725.0,11925.0

cmsbuild · 2021-04-12T21:21:29Z

-1

Failed Tests: Build Python3
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-914931/14200/summary.html
COMMIT: 440b14c
CMSSW: CMSSW_11_3_X_2021-04-12-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33415/14200/install.sh to create a dev area with all the needed externals and cmssw changes.

Build

I found compilation warning when building: See details on the summary page.

#Python3

I found errors:

*** Error compiling 'src/HLTrigger/Configuration/python/CustomConfigs.py'...

forthommel · 2021-04-12T22:20:12Z

HLTrigger/Configuration/python/CustomConfigs.py

@@ -200,3 +200,8 @@ def L1XML(process,xmlFile=None):
    process.ESPreferL1TXML = cms.ESPrefer("L1TUtmTriggerMenuESProducer","L1TriggerMenu")

    return process
+
+def CTPPSRun2Geometry(process):
+    if hasattr(process,'ctppsGeometryESModule')


Suggested change

if hasattr(process,'ctppsGeometryESModule')

if hasattr(process,'ctppsGeometryESModule'):

It is also necessary to return process as well as actually use the customization. Using the commit at 1364f26 on top of this PR allows the following to complete without errors in a local test:

runTheMatrix.py -l 11725.0,11925.0,limited --ibeos

addOnTests -j 8

thansk @forthommel @christopheralanwest .
@Martin-Grunewald is Configuration/HLT/python/addOnTestsHLT.py automatically generated from HLTrigger/Configuration/test/cmsDriver.csh ? Do we need to change both manually?

They are separate, both need the change.

ok, I've added 1364f26

Martin-Grunewald · 2021-04-13T13:06:42Z

+1

perrotta · 2021-04-13T13:19:17Z

+1

Modified reco-related code is identical to what previously submitted with PPS: update of era modifiers #33250 and Re-enable ctppsProtons from recoCTPPSTask in Run-3 #33266, plus the additional GT updates from previous Add PPS geometry and simulation tags to GTs #33404, which allows running in the run3 setup, and the corresponding HLT customizations
PPS era modifiers got reorganized, and proton reconstruction is brought back into the run3 reconstruction chain
Jenkins tests pass and the effect on output reflects what expected from the bug fix mentioned above

silviodonato · 2021-04-13T13:30:50Z

test parameters:
workflow = 11725.0,11925.0

silviodonato · 2021-04-13T13:33:20Z

please test
this is a further technical check by using the workflow specific for testing PPS
@cms-sw/dqm-l2 this additional test will not change the comparison result

cmsbuild · 2021-04-13T16:35:12Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-914931/14213/summary.html
COMMIT: d7a53f4
CMSSW: CMSSW_11_3_X_2021-04-12-2300/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/33415/14213/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-914931/11725.0_GluGluTo2Jets_14TeV+2021+GluGluTo2Jets_M_300_2000_14TeV_Exhume_GenSim+Digi+Reco+HARVEST+ALCA
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-914931/11925.0_GluGluTo2Jets_14TeV+2021PU+GluGluTo2Jets_M_300_2000_14TeV_Exhume_GenSim+DigiPU+RecoPU+HARVESTPU+Nano

Summary:

No significant changes to the logs found
Reco comparison results: 40 differences found in the comparisons
DQMHistoTests: Total files compared: 38
DQMHistoTests: Total histograms compared: 2865526
DQMHistoTests: Total failures: 20
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2865484
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 37 files compared)
Checked 160 log files, 37 edm output root files, 38 DQM output files
TriggerResults: no differences found

silviodonato · 2021-04-13T16:38:37Z

@jfernan2 the DQM part of the PR is basically renaming ctpps_2016 to ctpps
in

DQM/CTPPS/python/ctppsDQM_cff.py
DQM/CTPPS/test/all_ctpps_dqm_test_from_aod_cfg.py
DQM/CTPPS/test/all_ctpps_dqm_test_from_dat_cfg.py 
DQM/CTPPS/test/all_ctpps_dqm_test_from_raw_cfg.py

jfernan2 · 2021-04-13T18:00:49Z

+1

cmsbuild · 2021-04-13T18:01:11Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

silviodonato · 2021-04-13T19:04:48Z

+1

jan-kaspar and others added 9 commits March 19, 2021 09:22

add ctpps era modifier

7b10f04

update mapping between ctpps modifiers and Run2_20XY chains

2da85c9

replace ctpps_2016 with ctpps modifier, where desirable

518d281

re-enable pps reco in run3

ae7170c

Add PPS geometry and simulation tags to GTs

f35acf8

Merged alca-PPStags from repository malbouis with cms-merge-topic

185cd0e

Merged reenablePPSrun3 from repository silviodonato with cms-merge-topic

9796ddb

Merged pps_era_modifiers from repository CTPPS with cms-merge-topic

c602a7e

use CTPPSRun2Geometry customization to fix HLT test with Run2 data

440b14c

cmsbuild added this to the CMSSW_11_3_X milestone Apr 12, 2021

cmsbuild added alca-pending code-checks-pending dqm-pending hlt-pending operations-pending orp-pending pending-signatures reconstruction-pending tests-started labels Apr 12, 2021

silviodonato marked this pull request as draft April 12, 2021 20:56

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 12, 2021

cmsbuild added tests-rejected and removed tests-started labels Apr 12, 2021

forthommel reviewed Apr 12, 2021

View reviewed changes

cmsbuild added the alca-approved label Apr 13, 2021

This was referenced Apr 13, 2021

Add PPS geometry and simulation tags to GTs #33404

Merged

Re-enable ctppsProtons from recoCTPPSTask in Run-3 #33266

Merged

PPS: update of era modifiers #33250

Closed

cmsbuild added hlt-approved and removed hlt-pending labels Apr 13, 2021

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Apr 13, 2021

cmsbuild added tests-started and removed tests-approved labels Apr 13, 2021

cmsbuild added tests-approved and removed tests-started labels Apr 13, 2021

cmsbuild added dqm-approved fully-signed and removed dqm-pending pending-signatures labels Apr 13, 2021

cmsbuild added orp-approved and removed orp-pending labels Apr 13, 2021

cmsbuild merged commit 513b9c6 into cms-sw:master Apr 13, 2021

qliphy mentioned this pull request Apr 22, 2021

re-enable ctppsProtons for Run3 after #32352 #32356

Closed

slava77 mentioned this pull request Jul 22, 2021

PPS: update of era modifiers #33080

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

PPS: add Run-3 reco, add geometry to GTs, update era modifier #33415

PPS: add Run-3 reco, add geometry to GTs, update era modifier #33415

silviodonato commented Apr 12, 2021 •

edited

silviodonato commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

silviodonato commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

forthommel Apr 12, 2021

christopheralanwest Apr 13, 2021

silviodonato Apr 13, 2021

Martin-Grunewald Apr 13, 2021

silviodonato Apr 13, 2021

Martin-Grunewald commented Apr 13, 2021

perrotta commented Apr 13, 2021

silviodonato commented Apr 13, 2021

silviodonato commented Apr 13, 2021

cmsbuild commented Apr 13, 2021

silviodonato commented Apr 13, 2021

jfernan2 commented Apr 13, 2021

cmsbuild commented Apr 13, 2021

silviodonato commented Apr 13, 2021

	if hasattr(process,'ctppsGeometryESModule')
	if hasattr(process,'ctppsGeometryESModule'):

PPS: add Run-3 reco, add geometry to GTs, update era modifier #33415

PPS: add Run-3 reco, add geometry to GTs, update era modifier #33415

Conversation

silviodonato commented Apr 12, 2021 • edited

silviodonato commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

silviodonato commented Apr 12, 2021

cmsbuild commented Apr 12, 2021

Build

forthommel Apr 12, 2021

Choose a reason for hiding this comment

christopheralanwest Apr 13, 2021

Choose a reason for hiding this comment

silviodonato Apr 13, 2021

Choose a reason for hiding this comment

Martin-Grunewald Apr 13, 2021

Choose a reason for hiding this comment

silviodonato Apr 13, 2021

Choose a reason for hiding this comment

Martin-Grunewald commented Apr 13, 2021

perrotta commented Apr 13, 2021

silviodonato commented Apr 13, 2021

silviodonato commented Apr 13, 2021

cmsbuild commented Apr 13, 2021

Comparison Summary

silviodonato commented Apr 13, 2021

jfernan2 commented Apr 13, 2021

cmsbuild commented Apr 13, 2021

silviodonato commented Apr 13, 2021

silviodonato commented Apr 12, 2021 •

edited