reMiniAOD relval workflows for heavy ions #30354

mandrenguyen · 2020-06-24T08:11:36Z

Adds a new workflows 140.5611 and 158.01, to test reMiniAOD in data and MC, respectively.
Requested in #30349

cmsbuild · 2020-06-24T08:12:05Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-06-24T08:18:21Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-30354/16376

This PR adds an extra 56KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/relval_standard.py modified in PR(s): add wf 136.72411 re-miniAOD Run2 UL workflow for Run2016B useful for PPS testing #30351, HCAL only wf #30136
- File Configuration/PyReleaseValidation/python/relval_steps.py modified in PR(s): add wf 136.72411 re-miniAOD Run2 UL workflow for Run2016B useful for PPS testing #30351, HCAL only wf #30136

cmsbuild · 2020-06-24T08:18:47Z

A new Pull Request was created by @mandrenguyen (Matthew Nguyen) for master.

It involves the following packages:

Configuration/PyReleaseValidation
RecoTauTag/Configuration

@perrotta, @pgunnell, @chayanit, @wajidalikhan, @kpedro88, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @fabiocos, @slomeo this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

perrotta · 2020-06-24T08:55:14Z

please test workflow 159.11

cmsbuild · 2020-06-24T08:55:38Z

The tests are being triggered in jenkins.
Test Parameters:

MATRIX_EXTRAS = 159.11
CMSSW_11_2_X_2020-06-23-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7308/console Started: 2020/06/24 11:08

cmsbuild · 2020-06-24T10:22:43Z

+1
Tested at: 784c934
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e6456f/7308/summary.html
CMSSW: CMSSW_11_2_X_2020-06-23-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-24T10:22:46Z

Comparison job queued.

slava77 · 2020-06-24T11:54:49Z

Configuration/PyReleaseValidation/python/relval_steps.py

@@ -704,6 +704,8 @@ def identitySim(wf):
 # INPUT command for reminiAOD wfs on UL-like relval inputs
 steps['TTbar_13_reminiaod2016UL_preVFP_INPUT']={'INPUT':InputInfo(dataSet='/RelValTTbar_13UP16/CMSSW_10_6_12-PU25ns_106X_mcRun2_asymptotic_preVFP_v8_hltul16_preVFP-v1/AODSIM',label='rmaod',location='STD')}
 steps['TTbar_13_reminiaod2016UL_postVFP_INPUT']={'INPUT':InputInfo(dataSet='/RelValTTbar_13UP16/CMSSW_10_6_12-PU25ns_106X_mcRun2_asymptotic_v13_hltul16_postVFP-v1/AODSIM',label='rmaod',location='STD')}
+# INPUT command for reminiAOD wfs on PbPb relval inputs 
+steps['HydjetQ_reminiaodPbPb2021_INPUT']={'INPUT':InputInfo(dataSet='/RelValHydjetQ_B12_5020GeV_2021_ppReco/CMSSW_11_2_0_pre1-PU_111X_mcRun3_2021_realistic_HI_v5-v1/GEN-SIM-RECO',label='rmaod',location='STD')}


My goal from #30349 was to provide a reminiAOD setup using the target input dataset, which in this case would be from 10_3_X.
This one is apparently from 11_2_0, which makes this a bit impractical for the needs of reminiAOD

Ok, I assumed we would just do then when backporting to 10_6_X. But ok, we can run on a 10_3 input in the master. Will just have to fix the puppi issue.

I can input instead the dataset: /RelValHydjetQ_B12_5020GeV_2018_ppReco/CMSSW_10_3_3-PU_103X_upgrade2018_realistic_HI_v11-v1/GEN-SIM-RECO
However, it's only at T1_US_FNAL_Disk.
What does one enter in the function InputInfo, for the argument 'location' instead of 'STD'?
@chayanit maybe you can give me a pointer here?

is there an AOD file instead, to stay as close as possible to the (re)miniAOD scope?

No, it seems we are still writing RECOSIM in all the relval workflows.

cmsbuild · 2020-06-24T12:29:43Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-e6456f/7308/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-e6456f/159.11_HydjetQ_reminiaodPbPb2021_INPUT+HydjetQ_reminiaodPbPb2021_INPUT+REMINIAODHI2021PPRECO+HARVESTHI2021PPRECOMINIAOD

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2778811
DQMHistoTests: Total failures: 4
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2778757
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

slava77 · 2020-06-24T12:55:29Z

Adds a new MC workflow, 159.11, to test reMiniAOD in MC.

I understood that the (re)miniAOD will apply also to data.
Please add a workflow/setup for data as well.

cmsbuild · 2020-06-30T13:52:55Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #30237
Test Parameters:

MATRIX_EXTRAS = 140.5611

cmsbuild · 2020-06-30T13:53:28Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #30255
Test Parameters:

MATRIX_EXTRAS = 140.5611

cmsbuild · 2020-06-30T13:53:39Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #30313
Test Parameters:

MATRIX_EXTRAS = 140.5611

cmsbuild · 2020-06-30T15:35:45Z

The tests are being triggered in jenkins.
Tested with other pull request(s) #30253
Test Parameters:

MATRIX_EXTRAS = 140.5611
CMSSW_11_2_X_2020-06-30-1100/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/7531/console Started: 2020/06/30 17:39

cmsbuild · 2020-06-30T17:09:33Z

+1
Tested at: e89a9ce
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ec29ab/7531/summary.html
CMSSW: CMSSW_11_2_X_2020-06-30-1100
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-06-30T17:09:37Z

Comparison job queued.

cmsbuild · 2020-06-30T19:00:21Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-ec29ab/7531/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 36
DQMHistoTests: Total histograms compared: 2778915
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2778864
DQMHistoTests: Total skipped: 50
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 35 files compared)
Checked 152 log files, 16 edm output root files, 36 DQM output files

silviodonato · 2020-06-30T19:28:56Z

merge

santocch · 2020-07-10T07:41:37Z

+1

cmsbuild · 2020-07-10T07:42:05Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will be automatically merged.

slava77 · 2020-07-24T14:11:22Z

The heavier wf 140.5611 takes 4.4 s/event on my nominal 10HS06 node, compared to 2.1 s/ev in wf 136.88811 (reminiAOD of UL 2018 data).

Most of the heavy modules with dubious purpose for HI are PUPPI-related (taking >=20ms cases):
227.15 ms/ev patMETsPuppi
226.99 ms/ev patPFMetPuppi
507.88 ms/ev pfImpactParameterTagInfosPuppi
26.96 ms/ev packedCandidateTrackValidatorLostTracks
142.41 ms/ev pfSecondaryVertexTagInfosPuppi
42.28 ms/ev pfCombinedMVAV2BJetTagsPuppi
31.83 ms/ev pfCombinedCvsBJetTagsPuppi
39.85 ms/ev pfCombinedCvsLJetTagsPuppi
772.81 ms/ev puppi
771.44 ms/ev puppiNoLep
63.76 ms/ev ak4PFJetsPuppi
these add up to 2.85 s/ev and I suppose the next step will be to either crank up the thresholds or remove them from the sequences.

@mandrenguyen
this was mentioned in the RECO meeting earlier today

reMiniAOD wf for heavy ions

784c934

cmsbuild added this to the CMSSW_11_2_X milestone Jun 24, 2020

cmsbuild added code-checks-pending comparison-pending orp-pending pdmv-pending pending-signatures reconstruction-pending tests-pending upgrade-pending labels Jun 24, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 24, 2020

cmsbuild added tests-started and removed tests-pending labels Jun 24, 2020

cmsbuild added tests-approved and removed tests-started labels Jun 24, 2020

slava77 reviewed Jun 24, 2020

View reviewed changes

cmsbuild added comparison-available and removed comparison-pending labels Jun 24, 2020

Added a real data workflow, 140.61

d597b36

cmsbuild removed the code-checks-approved label Jun 24, 2020

cmsbuild added tests-pending tests-started and removed requires-external tests-pending labels Jun 30, 2020

cmsbuild added the requires-external label Jun 30, 2020

slava77 mentioned this pull request Jun 30, 2020

Add Pixel Tracks in Heavy Ions MiniAOD #30313

Merged

cmsbuild added tests-approved and removed tests-started labels Jun 30, 2020

cmsbuild added comparison-available and removed comparison-pending labels Jun 30, 2020

cmsbuild added orp-approved and removed orp-pending labels Jun 30, 2020

cmsbuild merged commit 4d72399 into cms-sw:master Jun 30, 2020

slava77 mentioned this pull request Jul 2, 2020

add re-miniAOD workflow for HI 2018 data #30349

Closed

cmsbuild added analysis-approved fully-signed and removed analysis-pending pending-signatures labels Jul 10, 2020

mandrenguyen deleted the hiReMiniAODRelValWF branch October 8, 2020 08:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

reMiniAOD relval workflows for heavy ions #30354

reMiniAOD relval workflows for heavy ions #30354

mandrenguyen commented Jun 24, 2020 •

edited

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

perrotta commented Jun 24, 2020

cmsbuild commented Jun 24, 2020 •

edited

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

slava77 Jun 24, 2020

mandrenguyen Jun 24, 2020

mandrenguyen Jun 24, 2020

slava77 Jun 24, 2020

mandrenguyen Jun 24, 2020

cmsbuild commented Jun 24, 2020

slava77 commented Jun 24, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020 •

edited

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

silviodonato commented Jun 30, 2020

santocch commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

slava77 commented Jul 24, 2020

reMiniAOD relval workflows for heavy ions #30354

reMiniAOD relval workflows for heavy ions #30354

Conversation

mandrenguyen commented Jun 24, 2020 • edited

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

perrotta commented Jun 24, 2020

cmsbuild commented Jun 24, 2020 • edited

cmsbuild commented Jun 24, 2020

cmsbuild commented Jun 24, 2020

slava77 Jun 24, 2020

Choose a reason for hiding this comment

mandrenguyen Jun 24, 2020

Choose a reason for hiding this comment

mandrenguyen Jun 24, 2020

Choose a reason for hiding this comment

slava77 Jun 24, 2020

Choose a reason for hiding this comment

mandrenguyen Jun 24, 2020

Choose a reason for hiding this comment

cmsbuild commented Jun 24, 2020

slava77 commented Jun 24, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020 • edited

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

cmsbuild commented Jun 30, 2020

silviodonato commented Jun 30, 2020

santocch commented Jul 10, 2020

cmsbuild commented Jul 10, 2020

slava77 commented Jul 24, 2020

mandrenguyen commented Jun 24, 2020 •

edited

cmsbuild commented Jun 24, 2020 •

edited

cmsbuild commented Jun 30, 2020 •

edited