Add tracking only PU workflows #39722

AdrianoDee · 2022-10-13T09:59:07Z

PR description:

This PR adds to the matrix the trackingOnly wfs with PU.

PR validation:

Running 11834.1 and 21034.1.

cmsbuild · 2022-10-13T10:06:28Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-39722/32557

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): MTD geometry: update BTL description #39670, Add run3 tracking low pu era #33532, [Draft] Update HGCAL pset for Phase-2 HLT menu #39450

cmsbuild · 2022-10-13T10:06:52Z

A new Pull Request was created by @AdrianoDee for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)

@bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen, @sunilUIET can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.

cms-bot commands are listed here

AdrianoDee · 2022-10-13T10:08:02Z

test parameters:

workflows = 11834.1, 22224.1
relvals_opt= -w upgrade

AdrianoDee · 2022-10-13T10:08:19Z

please test

cmsbuild · 2022-10-13T12:41:32Z

-1

Failed Tests: RelVals
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebe618/28220/summary.html
COMMIT: 6dec60d
CMSSW: CMSSW_12_6_X_2022-10-12-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39722/28220/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

22224.1

----- Begin Fatal Exception 13-Oct-2022 13:22:08 CEST-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

makortel · 2022-10-13T13:45:58Z

Just a historical note, when introducing the trackingOnly workflow I did not bother to add a version with pileup because for local use it was easy to use runTheMatrix.py to generate the configuration and change the input files manually to the desired ones (that could include pileup). Back then the purpose of having the workflow run in IBs was merely to ensure the workflow staying functional.

AdrianoDee · 2022-10-13T16:32:52Z

Indeed, I would add it to the release since I've received some requests about PU wf running tracking only that could be used for developments and it could ease things up.

Anyway, I've tested the 22224.0 (that should have not been touched by this release) locally and it fails with the same error. Seems more a problem of the wf itself that I have to understand. In the meanwhile I'd restart the tests with the D88 one, TuneCP5 that was the one I actually intended to test.

AdrianoDee · 2022-10-13T16:34:23Z

test parameters:

workflows = 11834.1, 21034.1
relvals_opt= -w upgrade

AdrianoDee · 2022-10-13T16:35:20Z

please test

cmsbuild · 2022-10-13T20:40:26Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebe618/28241/summary.html
COMMIT: 6dec60d
CMSSW: CMSSW_12_6_X_2022-10-13-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39722/28241/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/11834.1_TTbar_14TeV+2021PU_trackingOnly+TTbar_14TeV_TuneCP5_GenSim+DigiPU+RecoNanoPU+HARVESTNanoPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/20834.0_TTbar_14TeV+2026D88+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/20834.75_TTbar_14TeV+2026D88_HLT75e33+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HLT75e33+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/20896.0_CloseByPGun_CE_E_Front_120um+2026D88+CE_E_Front_120um_GenSimHLBeamSpotHGCALCloseBy+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/20900.0_CloseByPGun_CE_H_Coarse_Scint+2026D88+CE_H_Coarse_Scint_GenSimHLBeamSpotHGCALCloseBy+DigiTrigger+RecoGlobal+HARVESTGlobal
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/21034.1_TTbar_14TeV+2026D88PU_trackingOnly+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/21034.999_TTbar_14TeV+2026D88PU_PMXS1S2PR+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/23234.0_TTbar_14TeV+2026D94+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3392309
DQMHistoTests: Total failures: 0
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3392287
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 204 log files, 49 edm output root files, 49 DQM output files
TriggerResults: no differences found

srimanob · 2022-10-14T02:24:43Z

+Upgrade

This PR add PU for tracking only workflows. Tested workflows, 11834.1, 21034.1, run fine.

srimanob · 2022-10-25T15:34:49Z

Kindly ping @cms-sw/pdmv-l2

srimanob · 2022-10-25T15:35:11Z

@cmsbuild please test

Retrigger the test after almost 2 weeks.

cmsbuild · 2022-10-25T19:23:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ebe618/28497/summary.html
COMMIT: 6dec60d
CMSSW: CMSSW_12_6_X_2022-10-25-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/39722/28497/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/11834.1_TTbar_14TeV+2021PU_trackingOnly+TTbar_14TeV_TuneCP5_GenSim+DigiPU+RecoNanoPU+HARVESTNanoPU
/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-ebe618/21034.1_TTbar_14TeV+2026D88PU_trackingOnly+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
Reco comparison had 6 failed jobs
DQMHistoTests: Total files compared: 48
DQMHistoTests: Total histograms compared: 3378384
DQMHistoTests: Total failures: 3
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3378359
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 47 files compared)
Checked 201 log files, 48 edm output root files, 48 DQM output files
TriggerResults: no differences found

mmusich · 2022-11-10T18:23:24Z

just to understand what's still pending here? @cms-sw/pdmv-l2

kskovpen · 2022-11-10T19:15:25Z

+pdmv

cmsbuild · 2022-11-10T19:15:48Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @rappoccio (and backports should be raised in the release meeting by the corresponding L2)

rappoccio · 2022-11-10T21:29:30Z

+1

Add tracking only PU workflows

6dec60d

cmsbuild added this to the CMSSW_12_6_X milestone Oct 13, 2022

cmsbuild added code-checks-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Oct 13, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 13, 2022

cmsbuild added tests-started and removed tests-pending labels Oct 13, 2022

cmsbuild added tests-rejected and removed tests-started labels Oct 13, 2022

cmsbuild added tests-started and removed tests-rejected labels Oct 13, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 13, 2022

cmsbuild removed the upgrade-pending label Oct 14, 2022

cmsbuild added the upgrade-approved label Oct 14, 2022

cmsbuild mentioned this pull request Oct 14, 2022

Update Phase-2 Tracker and HGCAL HLT config files #39733

Merged

cmsbuild mentioned this pull request Oct 25, 2022

Add D97 Phase-2 geometry #39841

Merged

cmsbuild added tests-started and removed tests-approved labels Oct 25, 2022

cmsbuild added tests-approved and removed tests-started labels Oct 25, 2022

This was referenced Oct 26, 2022

Add AlCaReco producers in phase-2 workflows #39858

Merged

add protection for AlCa step in several specialized phase-2 workflows #39917

Merged

This was referenced Nov 7, 2022

New Vertex Clustering in Blocks and Weighted Mean Estimator #39995

Merged

HLT_Mu17_Photon30_IsoCaloId_v trigger filters backport to CMSSW_12_4_X #40034

Closed

cmsbuild added fully-signed pdmv-approved and removed pending-signatures pdmv-pending labels Nov 10, 2022

cmsbuild added orp-approved and removed orp-pending labels Nov 10, 2022

cmsbuild merged commit 54a030c into cms-sw:master Nov 10, 2022

AdrianoDee deleted the wf_trackingonly_pu branch November 15, 2022 08:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add tracking only PU workflows #39722

Add tracking only PU workflows #39722

AdrianoDee commented Oct 13, 2022 •

edited

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

makortel commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022 •

edited

AdrianoDee commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

srimanob commented Oct 14, 2022

srimanob commented Oct 25, 2022

srimanob commented Oct 25, 2022

cmsbuild commented Oct 25, 2022

mmusich commented Nov 10, 2022

kskovpen commented Nov 10, 2022

cmsbuild commented Nov 10, 2022

rappoccio commented Nov 10, 2022

Add tracking only PU workflows #39722

Add tracking only PU workflows #39722

Conversation

AdrianoDee commented Oct 13, 2022 • edited

PR description:

PR validation:

cmsbuild commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

RelVals

makortel commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022 • edited

AdrianoDee commented Oct 13, 2022

AdrianoDee commented Oct 13, 2022

cmsbuild commented Oct 13, 2022

Comparison Summary

srimanob commented Oct 14, 2022

srimanob commented Oct 25, 2022

srimanob commented Oct 25, 2022

cmsbuild commented Oct 25, 2022

Comparison Summary

mmusich commented Nov 10, 2022

kskovpen commented Nov 10, 2022

cmsbuild commented Nov 10, 2022

rappoccio commented Nov 10, 2022

AdrianoDee commented Oct 13, 2022 •

edited

AdrianoDee commented Oct 13, 2022 •

edited