Phase2 workflow fixes #14165

kpedro88 · 2016-04-20T11:04:58Z

This is the promised "workflow fix" PR for Phase2. This is just meant to get some things nominally working before 8_1_0_pre3. Further reorganizations will come later.

Major changes:

local reco workflows now use full local reco instead of just tracker local reco
2023 workflows now use the Phase2 era
rename 2023dev -> 2023tilted

Various bug fixes/customizations/patches:

turn off useLumiInfoRunHeader in ecalMultiFitUncalibRecHit to fix ECAL bunchSpacingProducer exception
turn off all "recover" options in ecalRecHit to avoid use of EcalRecHitWorkerRecover, which has non-optional use of EcalEndcapGeometryRecord that causes an exception in Phase2 geometries (a better fix for this should come later)
additional ZDC customization to handle lack of DIGI2RAW/RAW2DIGI for HCAL
temporary fix from @jshlee to prevent ME0 crash

Known warnings:

Expected since HEback isn't set up in the hexagonal geometry yet:

%MSG-e HGCDigitizer:  MixingModule:mix 20-Apr-2016 02:57:06 CDT Run: 1 Event: 1
 @ accumulate : can't find HGCHitsHEback collection of g4SimHits

CSC people are looking into this one, the actual file address is /cvmfs/cms-ib.cern.ch/2016-17/slc6_amd64_gcc530/cms/CSCTrackFinderEmulation/1.2/data/L1Trigger/CSCTrackFinder/data/core_2014_05_15/comp_dphi_8.dat with a slash between data and L1Trigger:

Cannot read file: /cvmfs/cms-ib.cern.ch/2016-17/slc6_amd64_gcc530/cms/CSCTrackFinderEmulation/1.2/dataL1Trigger/CSCTrackFinder/data/core_2014_05_15/comp_dphi_8.dat, addr: 0

I tested this using the normal and tilted FourMuPt1_200 upgrade workflows:

runTheMatrix.py -w upgrade -l 11000.0 > & matrix_11000.log &
runTheMatrix.py -w upgrade -l 10600.0 > & matrix_10600.log &

10 events run to completion for steps 1 and 2 with no crashes or exceptions.

attn: @boudoul, @lgray, @bsunanda, @ianna, @calabria

@davidlange6 - should we add upgrade workflows into the PR/IB standard tests in this PR, or wait?

cmsbuild · 2016-04-20T11:05:19Z

A new Pull Request was created by @kpedro88 (Kevin Pedro) for CMSSW_8_1_X.

It involves the following packages:

Configuration/Geometry
Configuration/PyReleaseValidation
Configuration/StandardSequences
Geometry/CMSCommonData
RecoLocalCalo/Configuration
RecoLocalCalo/EcalRecProducers
RecoLocalMuon/GEMSegment
SLHCUpgradeSimulations/Configuration
SLHCUpgradeSimulations/Geometry

@civanch, @Dr15Jones, @cvuosalo, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @srimanob, @franzoni, @slava77, @hengne, @davidlange6 can you please review it and eventually sign? Thanks.
@ghellwig, @makortel, @GiacomoSguazzoni, @rovere, @VinInn, @Martin-Grunewald, @bellan, @jhgoh, @cerati, @argiro, @dgulhan this is something you requested to watch as well.
@slava77, @Degano, @smuzaffar you are the release manager for this.

cms-bot commands are list here #13028

kpedro88 · 2016-04-20T11:05:38Z

@cmsbuild please test

cmsbuild · 2016-04-20T11:05:56Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12517/console

slava77 · 2016-04-20T11:12:26Z

RecoLocalCalo/EcalRecProducers/python/ecalMultiFitUncalibRecHit_cfi.py

@@ -63,3 +63,6 @@
      chi2ThreshEE_ = cms.double(50.0),
   )                                           
 )
+
+from Configuration.StandardSequences.Eras import eras
+eras.phase2_common.toModify( ecalMultiFitUncalibRecHit.algoPSet, useLumiInfoRunHeader = cms.bool(False) )


The underlying problem is that bunchSpacingProducer is separate from localreco:
https://github.com/cms-sw/cmssw/blob/CMSSW_8_1_X/Configuration/StandardSequences/python/Reconstruction_cff.py#L101

Without it, there is an exception when running ecalMultiFitUncalibRecHit. The above customization seemed like the easiest way to prevent the exception. I'm open to other solutions.

add it to the sequence you are running (at the start).
It's already in the localreco in HI and cosmics.
There is no harm from listing the same module several times in different sequences

cmsbuild · 2016-04-20T12:40:36Z

-1

Tested at: e26834b

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14165/12517/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

RelVals:

When I ran the RelVals I found an error in the following worklfows:
140.53 step2

runTheMatrix-results/140.53_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI/step2_RunHI2011+RunHI2011+RECOHID11+HARVESTDHI.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4.log

1003.0 step2

runTheMatrix-results/1003.0_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM/step2_RunMinBias2012A+RunMinBias2012A+RECODDQM+HARVESTDDQM.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:15:21 2016-date Wed Apr 20 14:08:39 2016 s - exit: 16640
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:17:15 2016-date Wed Apr 20 14:08:40 2016 s - exit: 16640
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:15:46 2016-date Wed Apr 20 14:09:53 2016 s - exit: 16640
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:20:59 2016-date Wed Apr 20 14:18:06 2016 s - exit: 16640
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --magField 38T_PostLS1 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:25:52 2016-date Wed Apr 20 14:19:30 2016 s - exit: 16640
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Wed Apr 20 14:24:40 2016-date Wed Apr 20 14:21:04 2016 s - exit: 16640

boudoul · 2016-04-20T12:52:47Z

looks like the errors are not related to this PR (other PRs are suffering from the same errors ..)

kpedro88 · 2016-04-20T12:54:06Z

@boudoul - yes, I've seen them in other PRs also. It appears to be a problem with jet corrections:

An exception of category 'NoRecord' occurred while
   [0] Processing run: 165121 lumi: 62 event: 23609118
   [1] Running path 'reconstruction_step'
   [2] Calling event method for module L1FastjetCorrectorProducer/'ak4PFL1FastjetCorrector'
Exception Message:
No "JetCorrectionsRecord" record found in the EventSetup for synchronization value
Run: 165121 LuminosityBlock: 62 Event: 0 Time: 5607255048184066530
 Please add an ESSource or ESProducer that delivers such a record.

hengne · 2016-04-20T13:00:52Z

@boudoul @kpedro88 it is xrootd access to the input files.. my PR is also suffering this since yesterday evening: #14098

cmsbuild · 2016-04-20T13:09:17Z

Pull request #14165 was updated. @civanch, @Dr15Jones, @cvuosalo, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @srimanob, @franzoni, @slava77, @hengne, @davidlange6 can you please check and sign again.

kpedro88 · 2016-04-20T13:09:43Z

@slava77 latest commit addresses your recommendation about bunchSpacingProducer

slava77 · 2016-04-20T13:28:40Z

Configuration/StandardSequences/python/Reconstruction_cff.py

@@ -194,4 +196,5 @@
 #
 reconstruction_standard_candle = cms.Sequence(localreco*globalreco*vertexreco*recoJetAssociations*btagging*electronSequence*photonSequence)

-
+from Configuration.StandardSequences.Eras import eras
+eras.phase2_common.toReplaceWith( localreco, _phase2_localreco )


maybe easier to add globally, no eras

is that allowed? I don't want to screw up other workflows

should be OK, we'll see in the tests anyways.

submitted as a separate commit so it can be reverted back to the Era-dependent version if necessary

kpedro88 · 2016-04-20T13:46:13Z

Update on known warning 2: it's unrelated to Phase2, caused by a change in the external. I submitted an issue against the external: cms-externals/CSCTrackFinderEmulation#3

cmsbuild · 2016-04-20T13:47:33Z

Pull request #14165 was updated. @civanch, @Dr15Jones, @cvuosalo, @ianna, @mdhildreth, @fabozzi, @cmsbuild, @srimanob, @franzoni, @slava77, @hengne, @davidlange6 can you please check and sign again.

smuzaffar · 2016-04-20T13:54:50Z

@kpedro88 , cms-externals/CSCTrackFinderEmulation tool file is updated to have the correct path. Next IB (23h00) should have the fix.

kpedro88 · 2016-04-20T13:55:42Z

@smuzaffar great, thanks

kpedro88 · 2016-04-20T20:55:01Z

@cmsbuild please test

cmsbuild · 2016-04-20T20:55:21Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/12530/console

cmsbuild · 2016-04-20T23:40:46Z

+1
Tested at: 5dda382
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14165/12530/summary.html

cmsbuild · 2016-04-21T00:26:57Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-14165/12530/summary.html

kpedro88 · 2016-04-21T07:13:50Z

The comparison looks good to me.

@davidlange6 @slava77 @civanch @ianna @hengne - can you sign ASAP so we can make 810pre3?

davidlange6 · 2016-04-21T11:50:04Z

all - will merge this tonight as to build pre3 unless there are issues raised in the meantime

Add phase ii scenarios to runTheMatrix on top of #14098 #14165

kpedro88 · 2016-04-21T18:57:13Z

@davidlange6 did you mean to merge this?

kpedro88 · 2016-04-21T18:59:21Z

Ah I see, it was included in #14181

kpedro88 added 5 commits April 20, 2016 04:38

enable full localreco and Phase2 era in upgrade workflows

b8a1e0a

additional zdc Phase2 custom

ebe25a3

ECAL customs to fix exceptions in Phase2 workflows

c955e36

rename 2023dev -> 2023tilted

ec4b9a0

fix from @jshlee for ME0 crash

e26834b

cmsbuild added this to the Next CMSSW_8_1_X milestone Apr 20, 2016

cmsbuild added reconstruction-pending simulation-pending geometry-pending operations-pending pending-signatures tests-pending orp-pending comparison-pending pdmv-pending labels Apr 20, 2016

cmsbuild added tests-started and removed tests-pending labels Apr 20, 2016

slava77 reviewed Apr 20, 2016
View reviewed changes

cmsbuild added tests-rejected and removed tests-started labels Apr 20, 2016

kpedro88 mentioned this pull request Apr 20, 2016

HGCal Eras using recommended methods #14168

Merged

bunchSpacingProducer in localreco for phase2

af926e9

cmsbuild added tests-pending and removed tests-rejected labels Apr 20, 2016

slava77 reviewed Apr 20, 2016
View reviewed changes

hengne mentioned this pull request Apr 20, 2016

relval matrix update + data relval HLT:@relval2016 #14098

Merged

bunchSpacingProducer in localreco for all

5dda382

cmsbuild added tests-started and removed tests-pending labels Apr 20, 2016

cmsbuild added tests-approved and removed tests-started labels Apr 20, 2016

cmsbuild added comparison-available and removed comparison-pending labels Apr 21, 2016

davidlange6 closed this Apr 21, 2016

davidlange6 added a commit that referenced this pull request Apr 21, 2016

Merge pull request #14181 from davidlange6/addPhIIscenarios

692264b

Add phase ii scenarios to runTheMatrix on top of #14098 #14165

kpedro88 mentioned this pull request May 2, 2016

remove unneeded EE geometry from EcalRecHitWorkerRecover #14333

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Phase2 workflow fixes #14165

Phase2 workflow fixes #14165

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

slava77 Apr 20, 2016

kpedro88 Apr 20, 2016

slava77 Apr 20, 2016

cmsbuild commented Apr 20, 2016

boudoul commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

hengne commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

slava77 Apr 20, 2016

kpedro88 Apr 20, 2016

slava77 Apr 20, 2016

kpedro88 Apr 20, 2016

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

smuzaffar commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

cmsbuild commented Apr 21, 2016

kpedro88 commented Apr 21, 2016

davidlange6 commented Apr 21, 2016

kpedro88 commented Apr 21, 2016

kpedro88 commented Apr 21, 2016

Phase2 workflow fixes #14165

Phase2 workflow fixes #14165

Conversation

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

slava77 Apr 20, 2016

Choose a reason for hiding this comment

kpedro88 Apr 20, 2016

Choose a reason for hiding this comment

slava77 Apr 20, 2016

Choose a reason for hiding this comment

cmsbuild commented Apr 20, 2016

boudoul commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

hengne commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

slava77 Apr 20, 2016

Choose a reason for hiding this comment

kpedro88 Apr 20, 2016

Choose a reason for hiding this comment

slava77 Apr 20, 2016

Choose a reason for hiding this comment

kpedro88 Apr 20, 2016

Choose a reason for hiding this comment

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

smuzaffar commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

kpedro88 commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

cmsbuild commented Apr 20, 2016

cmsbuild commented Apr 21, 2016

kpedro88 commented Apr 21, 2016

davidlange6 commented Apr 21, 2016

kpedro88 commented Apr 21, 2016

kpedro88 commented Apr 21, 2016