fix `VectorHits` workflows with PU #38468

mmusich · 2022-06-22T13:23:34Z

PR description:

it was noticed by @JanFSchulte that running wf 39634.9 it was crashing on the harvesting step with a seg fault in Phase2ITRecHitHarvester:Phase2OTTrackingRechitHarvester_PS.
This happens because this protection:

cmssw/Validation/SiTrackerPhase2V/python/Phase2TrackerMCHarvesting_cff.py

Lines 70 to 71 in c3498e8

    
           from Configuration.ProcessModifiers.vectorHits_cff import vectorHits 
        
           vectorHits.toReplaceWith(trackerphase2ValidationHarvesting, trackerphase2ValidationHarvesting.copyAndExclude([Phase2OTTrackingRechitHarvester_PS,Phase2OTTrackingRechitHarvester_2S]))

was not picked up, as the modifier was not applied in the harvesting step.
This PR trivially fixes the problem.

PR validation:

Run the wf 39634.9 and I get:

39634.9 2026D88PU_vectorHits+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU [1]: cmsDriver.py TTbar_14TeV_TuneCP5_cfi  -s GEN,SIM -n 10 --conditions auto:phase2_realistic_T21 --beamspot HLLHC14TeV --datatier GEN-SIM --eventcontent FEVTDEBUG --geometry Extended2026D88 --era Phase2C17I13M9 --relval 9000,100 
                                           [2]: cmsDriver.py step2  -s DIGI:pdigi_valid,L1TrackTrigger,L1,DIGI2RAW,HLT:@fake2 --conditions auto:phase2_realistic_T21 --datatier GEN-SIM-DIGI-RAW -n 10 --eventcontent FEVTDEBUGHLT --geometry Extended2026D88 --era Phase2C17I13M9 --pileup AVE_200_BX_25ns --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM
                                           [3]: cmsDriver.py step3  -s RAW2DIGI,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --conditions auto:phase2_realistic_T21 --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO -n 10 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM --geometry Extended2026D88 --era Phase2C17I13M9 --procModifiers vectorHits --pileup AVE_200_BX_25ns --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM
                                           [4]: cmsDriver.py step4  -s HARVESTING:@phase2Validation+@phase2+@miniAODValidation+@miniAODDQM --conditions auto:phase2_realistic_T21 --mc  --geometry Extended2026D88 --scenario pp --filetype DQM --era Phase2C17I13M9 --procModifiers vectorHits -n 10 --pileup AVE_200_BX_25ns --pileup_input das:/RelValMinBias_14TeV/CMSSW_12_3_0_pre5-123X_mcRun4_realistic_v4_2026D88noPU-v1/GEN-SIM

as expected (notice the modifier gets added to the step4).

if this PR is a backport please specify the original PR and why you need to backport that PR:

Not a backport but will be backported.

mmusich · 2022-06-22T13:24:03Z

type bug-fix

mmusich · 2022-06-22T13:26:25Z

test parameters:

workflows = 39634.9
relvals_opt= -w standard,highstats,pileup,generator,extendedgen,production,upgrade,cleanedupgrade,ged

cmsbuild · 2022-06-22T13:29:14Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-38468/30681

This PR adds an extra 24KB to repository
There are other open Pull requests which might conflict with changes you have proposed:
- File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): Add B physics wfs to release validation #38397, Add run3 tracking low pu era #33532

cmsbuild · 2022-06-22T13:29:35Z

A new Pull Request was created by @mmusich (Marco Musich) for master.

It involves the following packages:

Configuration/PyReleaseValidation (pdmv, upgrade)

@jordan-martins, @bbilin, @cmsbuild, @AdrianoDee, @srimanob, @kskovpen can you please review it and eventually sign? Thanks.
@makortel, @kpedro88, @fabiocos, @Martin-Grunewald, @missirol, @trtomei, @beaucero, @slomeo this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2022-06-22T13:35:46Z

please test

cmsbuild · 2022-06-22T17:29:42Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b0760/25700/summary.html
COMMIT: ce7d818
CMSSW: CMSSW_12_5_X_2022-06-22-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38468/25700/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 2 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659099
DQMHistoTests: Total failures: 7
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3659069
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 49 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2022-06-24T08:15:00Z

please test workflow 39634.9

apparently the command fix VectorHits workflows with PU #38468 (comment) wasn't picked up correctly (?)

cmsbuild · 2022-06-24T12:30:34Z

-1

Failed Tests: RelVals-INPUT
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b0760/25752/summary.html
COMMIT: ce7d818
CMSSW: CMSSW_12_5_X_2022-06-23-2300/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38468/25752/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals-INPUT

The relvals timed out after 4 hours.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659307
DQMHistoTests: Total failures: 14
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3659271
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2022-06-24T14:07:19Z

please test

cmsbuild · 2022-06-24T17:52:25Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b0760/25764/summary.html
COMMIT: ce7d818
CMSSW: CMSSW_12_5_X_2022-06-24-1100/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/38468/25764/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659307
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3659277
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 208 log files, 45 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2022-06-26T04:21:12Z

Hi @cms-sw/orp-l2, @smuzaffar,
I keep not seeing wf 39634.9 in https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b0760/25764/runTheMatrix-results/
so it seems that #38468 (comment) is not the right command to use. Can you clarify what command should be used?
Thanks you!

kpedro88 · 2022-06-26T12:37:01Z

@mmusich the problem is that you are using both "upgrade" and "cleanedupgrade". The latter exists so that the former does not have to be used, because it would cause duplicate workflow conflicts with "standard". Instead, using both of them together causes different duplicate workflow conflicts, as the relvals console log indicates:

ValueError: Duplicated workflows: 39634.9

The correct option is: relvals_opt= -w standard,highstats,pileup,generator,extendedgen,production,cleanedupgrade,ged

mmusich · 2022-06-26T15:34:02Z

test parameters:

workflow = 39634.9
relvals_opt= -w standard,highstats,pileup,generator,extendedgen,production,cleanedupgrade,ged

mmusich · 2022-06-26T15:34:13Z

@cmsbuild, please test

cmsbuild · 2022-06-26T20:05:28Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-9b0760/25809/summary.html
COMMIT: ce7d818
CMSSW: CMSSW_12_5_X_2022-06-26-0000/el8_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/38468/25809/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
39634.9 step 4
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-9b0760/39634.9_TTbar_14TeV+2026D88PU_vectorHits+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTriggerPU+RecoGlobalPU+HARVESTGlobalPU

Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 50
DQMHistoTests: Total histograms compared: 3659995
DQMHistoTests: Total failures: 8
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3659965
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 49 files compared)
Checked 212 log files, 49 edm output root files, 50 DQM output files
TriggerResults: no differences found

mmusich · 2022-06-27T07:59:17Z

@cms-sw/pdmv-l2 @cms-sw/upgrade-l2 now that the tests are finally proving that the Harvesting step for wf 39634.9 is running fine (see log), please consider signing this.
Thank you.

kskovpen · 2022-06-27T08:00:33Z

+pdmv

srimanob · 2022-06-27T08:04:10Z

+Upgrade

This PR just add HarvestGlobal to the vectorhits workflow.

cmsbuild · 2022-06-27T08:04:33Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-06-27T11:11:01Z

+1

fix VH workflows with PU

ce7d818

cmsbuild added this to the CMSSW_12_5_X milestone Jun 22, 2022

cmsbuild added code-checks-pending orp-pending pdmv-pending pending-signatures tests-pending upgrade-pending labels Jun 22, 2022

cmsbuild added the bug-fix label Jun 22, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Jun 22, 2022

cmsbuild added tests-started and removed tests-pending labels Jun 22, 2022

mmusich mentioned this pull request Jun 22, 2022

fix VectorHits workflows with PU [12.4.X] #38469

Merged

cmsbuild added tests-approved and removed tests-started labels Jun 22, 2022

cmsbuild mentioned this pull request Jun 23, 2022

HLT menu development for 12_4_X (4/N) + migration to 12_4_0 + frozen menu V1.2 for 2022 [12_5_X] #38482

Merged

cmsbuild added tests-started and removed tests-approved labels Jun 24, 2022

cmsbuild mentioned this pull request Jun 24, 2022

Adding Xi, Chi_c and Chi_b to the matrix for RelVals. #38494

Merged

cmsbuild added tests-rejected and removed tests-started labels Jun 24, 2022

cmsbuild added tests-started and removed tests-rejected labels Jun 24, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 24, 2022

cmsbuild added tests-started and removed tests-approved labels Jun 26, 2022

cmsbuild added tests-approved and removed tests-started labels Jun 26, 2022

cmsbuild added pdmv-approved and removed pdmv-pending labels Jun 27, 2022

cmsbuild added fully-signed upgrade-approved and removed pending-signatures upgrade-pending labels Jun 27, 2022

cmsbuild added orp-approved and removed orp-pending labels Jun 27, 2022

cmsbuild merged commit dfcb0e1 into cms-sw:master Jun 27, 2022

cmsbuild mentioned this pull request Jun 28, 2022

Set appropriate streamLabel for online DQM GPU validation #38527

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix `VectorHits` workflows with PU #38468

fix `VectorHits` workflows with PU #38468

mmusich commented Jun 22, 2022 •

edited

mmusich commented Jun 22, 2022

mmusich commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

mmusich commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

mmusich commented Jun 24, 2022

cmsbuild commented Jun 24, 2022

mmusich commented Jun 24, 2022

cmsbuild commented Jun 24, 2022

mmusich commented Jun 26, 2022

kpedro88 commented Jun 26, 2022

mmusich commented Jun 26, 2022

mmusich commented Jun 26, 2022

cmsbuild commented Jun 26, 2022

mmusich commented Jun 27, 2022

kskovpen commented Jun 27, 2022

srimanob commented Jun 27, 2022

cmsbuild commented Jun 27, 2022

qliphy commented Jun 27, 2022

	from Configuration.ProcessModifiers.vectorHits_cff import vectorHits
	vectorHits.toReplaceWith(trackerphase2ValidationHarvesting, trackerphase2ValidationHarvesting.copyAndExclude([Phase2OTTrackingRechitHarvester_PS,Phase2OTTrackingRechitHarvester_2S]))

fix VectorHits workflows with PU #38468

fix VectorHits workflows with PU #38468

Conversation

mmusich commented Jun 22, 2022 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR and why you need to backport that PR:

mmusich commented Jun 22, 2022

mmusich commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

mmusich commented Jun 22, 2022

cmsbuild commented Jun 22, 2022

Comparison Summary

mmusich commented Jun 24, 2022

cmsbuild commented Jun 24, 2022

RelVals-INPUT

Comparison Summary

mmusich commented Jun 24, 2022

cmsbuild commented Jun 24, 2022

Comparison Summary

mmusich commented Jun 26, 2022

kpedro88 commented Jun 26, 2022

mmusich commented Jun 26, 2022

mmusich commented Jun 26, 2022

cmsbuild commented Jun 26, 2022

Comparison Summary

mmusich commented Jun 27, 2022

kskovpen commented Jun 27, 2022

srimanob commented Jun 27, 2022

cmsbuild commented Jun 27, 2022

qliphy commented Jun 27, 2022

fix `VectorHits` workflows with PU #38468

fix `VectorHits` workflows with PU #38468

mmusich commented Jun 22, 2022 •

edited