add TestOldDigi workflow

[kpedro88]

PR description:

A new special upgrade workflow is created to allow testing of 11_0_X GEN-SIM-DIGI-RAW files in 11_1_X. This supports the compatibility needed for the planned HLT TDR re-reco campaign.

The new workflows 23234.1001, 23434.1001 (without and with PU) are forwarded to the regular matrix so they can be called on demand in PR tests.

Technically, this PR required a few additional modifications in relval_steps.py to allow PU-specific changes by special workflows, as well as improved handling of steps that are None (and therefore should be removed by the matrix read).

PR validation:

Ran workflow 23234.1001 locally and it succeeded.

attn: @fwyzard @trtomei

[cmsbuild]

The code-checks are being triggered in jenkins.

[cmsbuild]

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28775/13430

• This PR adds an extra 52KB to repository

• There are other open Pull requests which might conflict with changes you have proposed:

  • File Configuration/PyReleaseValidation/python/upgradeWorkflowComponents.py modified in PR(s): PyReleaseValidation production-like workflow for Run3 (2021) #28773

[cmsbuild]

A new Pull Request was created by @kpedro88 (Kevin Pedro) for master.

It involves the following packages:

Configuration/PyReleaseValidation

@chayanit, @cmsbuild, @pgunnell, @kpedro88, @zhenhu can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald this is something you requested to watch as well.
@davidlange6, @silviodonato, @fabiocos you are the release manager for this.

cms-bot commands are listed here

[fwyzard]

should we use the 11.0.0 relval, /RelValTTbar_14TeV/CMSSW_11_0_0-110X_mcRun4_realistic_v2_2026D49noPU-v2/GEN-SIM-DIGI-RAW ?

[kpedro88]

since there's no PU one (yet), I was just going to wait and update it later.

[fwyzard]

makes sense

[kpedro88]

test parameters
workflow = 23234.1001

[kpedro88]

please test

[cmsbuild]

The tests are being triggered in jenkins.
Test Parameters:

• MATRIX_EXTRAS = 23234.1001
  https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/4303/console Started: 2020/01/21 23:00

[cmsbuild]

+1
Tested at: 893500b
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8bdaa0/4303/summary.html
CMSSW: CMSSW_11_1_X_2020-01-21-1100
SCRAM_ARCH: slc7_amd64_gcc820

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-8bdaa0/4303/summary.html

@slava77 comparisons for the following workflows were not done due to missing matrix map:

• /data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/results/JR-comparison/PR-8bdaa0/23234.1001_TTbar_14TeV+RecoFullGlobal_TestOldDigi_2026D49+HARVESTFullGlobal_2026D49

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 0 differences found in the comparisons
• DQMHistoTests: Total files compared: 34
• DQMHistoTests: Total histograms compared: 2697090
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 2696743
• DQMHistoTests: Total skipped: 346
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
• Checked 147 log files, 16 edm output root files, 34 DQM output files

[kpedro88]

+upgrade

[chayanit]

+1

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @silviodonato, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

[silviodonato]

+1

[silviodonato]

Hi @kpedro88 ,
the new workflow 23434.1001 is crashing
https://cmssdt.cern.ch/SDT/html/cmssdt-ib/#/relVal/CMSSW_11_1/2020-01-22-1400?selectedArchs=slc7_amd64_gcc820&selectedFlavors=X&selectedStatus=failed

23-Jan-2020 09:27:32 CET  Successfully opened file root://xrootd-cms.infn.it//store/relval/CMSSW_11_0_0_pre13/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU25ns_110X_mcRun4_realistic_v2_2026D49PU200-v2/20000/427FE90B-760A-6F45-9B29-439CFA95A534.root
2020-01-23 09:28:15.512472: I tensorflow/core/platform/cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 AVX512F FMA
23-Jan-2020 09:28:52 CET  Closed file root://xrootd-cms.infn.it//store/relval/CMSSW_11_0_0_pre13/RelValTTbar_14TeV/GEN-SIM-DIGI-RAW/PU25ns_110X_mcRun4_realistic_v2_2026D49PU200-v2/20000/427FE90B-760A-6F45-9B29-439CFA95A534.root
----- Begin Fatal Exception 23-Jan-2020 09:28:52 CET-----------------------
An exception of category 'NoSecondaryFiles' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=MixingModule label='mix'
Exception Message:
RootEmbeddedFileSequence no input files specified for secondary input source.
----- End Fatal Exception -------------------------------------------------

the cmsDriver command is

cmsDriver.py step1  --conditions auto:phase2_realistic_T15 --pileup_input das:/RelValMinBias_14TeV/1/GEN-SIM -n 10 --era Phase2C9 --eventcontent FEVTDEBUGHLT,MINIAODSIM,DQM --runUnscheduled  --filein das:/RelValTTbar_14TeV/CMSSW_11_0_0_pre13-PU25ns_110X_mcRun4_realistic_v2_2026D49PU200-v2/GEN-SIM-DIGI-RAW -s RAW2DIGI,L1Reco,RECO,RECOSIM,PAT,VALIDATION:@phase2Validation+@miniAODValidation,DQM:@phase2+@miniAODDQM --datatier GEN-SIM-RECO,MINIAODSIM,DQMIO --pileup AVE_200_BX_25ns --geometry Extended2026D49  --customise Validation/Performance/TimeMemorySummary.customiseWithTimeMemorySummary --prefix '/data/cmsbld/jenkins/workspace/ib-run-relvals/cms-bot/monitor_workflow.py timeout --signal SIGTERM 9000 '  --fileout file:step1.root  --suffix "-j JobReport1.xml "  --nThreads 4 > step1_TTbar_14TeV+RecoFullGlobalPU_TestOldDigi_2026D49PU+HARVESTFullGlobalPU_2026D49PU.log  2>&1

I think the reason is --pileup_input das:/RelValMinBias_14TeV/1/GEN-SIM -n 10.
Could you fix it? Thanks!

[kpedro88]

@silviodonato see #28782