Failures in 23434.9921 and 34834.9921 workflows

[makortel]

Workflows 23434.9921 and 34834.9921 fail in step6 in CMSSW_12_1_X_2021-10-08-1100 with

----- Begin Fatal Exception 08-Oct-2021 19:31:20 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
   [0] Processing  Event run: 1 lumi: 7 event: 603 stream: 0
   [1] Running path 'NANOEDMAODSIMoutput_step'
   [2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
   [3] Prefetching for module GlobalVariablesTableProducer/'ttbarCategoryTable'
   [4] Prefetching for module GenTtbarCategorizer/'categorizeGenTtbar'
   [5] Calling method for module GenHFHadronMatcher/'matchGenCHadron'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetFlavourInfoMatchingCollection
Looking for module label: slimmedGenJetsFlavourInfos
Looking for productInstanceName: 

   Additional Info:
      [a] If you wish to continue processing events after a ProductNotFound exception,
add "SkipEvent = cms.untracked.vstring('ProductNotFound')" to the "options" PSet in the configuration.

----- End Fatal Exception -------------------------------------------------

https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_1_X_2021-10-08-1100/pyRelValMatrixLogs/run/23434.9921_TTbar_14TeV+2026D49PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU+NanoPU/step6_TTbar_14TeV+2026D49PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU+NanoPU.log#/
https://cmssdt.cern.ch/SDT/cgi-bin/logreader/slc7_amd64_gcc900/CMSSW_12_1_X_2021-10-08-1100/pyRelValMatrixLogs/run/34834.9921_TTbar_14TeV+2026D76PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU+NanoPU/step6_TTbar_14TeV+2026D76PU_PMXS1S2ProdLike+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14INPUT+PREMIX_PremixHLBeamSpot14PU+DigiTriggerPU+RecoGlobalPU+MiniAODPU+NanoPU.log#/

[cmsbuild]

A new Issue was created by @makortel Matti Kortelainen.

@Dr15Jones, @perrotta, @dpiparo, @makortel, @smuzaffar, @qliphy can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[makortel]

assign xpog, upgrade

[cmsbuild]

New categories assigned: upgrade,xpog

@mariadalfonso,@AdrianoDee,@gouskos,@srimanob,@fgolf you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

It is not immediately clear to me which of the PRs merged in CMSSW_12_1_X_2021-10-08-1100 could have caused this.

[qliphy]

It seems to be due to #35412
I could reproduce the error with CMSSW_12_1_X_2021-10-07-2300 + #35412

----- Begin Fatal Exception 09-Oct-2021 05:35:44 CEST-----------------------
An exception of category 'ProductNotFound' occurred while
[0] Processing Event run: 1 lumi: 1 event: 1 stream: 0
[1] Running path 'NANOEDMAODSIMoutput_step'
[2] Prefetching for module PoolOutputModule/'NANOEDMAODSIMoutput'
[3] Prefetching for module GlobalVariablesTableProducer/'ttbarCategoryTable'
[4] Prefetching for module GenTtbarCategorizer/'categorizeGenTtbar'
[5] Calling method for module GenHFHadronMatcher/'matchGenCHadron'
Exception Message:
Principal::getByToken: Found zero products matching all criteria
Looking for type: reco::JetFlavourInfoMatchingCollection
Looking for module label: slimmedGenJetsFlavourInfos
Looking for productInstanceName:

@kskovpen

[srimanob]

Thanks for reporting @makortel @qliphy

The issue comes from the wrong input to the nano step.

08-Oct-2021 19:29:50 CEST  Initiating request to open file file:step4.root
08-Oct-2021 19:29:53 CEST  Successfully opened file file:step4.root

Nano step handles inputs from either (1) Mini from the same step as RECO or (2) Mini as a separated step. So, we specify clearly in the Nano step which input should be picked. In this failure, it is the case (2). However, workflow .9921 have an additional step in the beginning to make a mixing library. So, input needs to shift by 1, step4 -> step5.

Here is the PR to fix:
#35597