Include "Scouting" packers and collections in MC production #21297

fwyzard · 2017-11-14T10:37:31Z

update EndPaths kept in a cff python dump: keep Scouting* producers, drop the Scouting OutputModules, and change the Scouting EndPaths into regular Paths;
keep the Scouting collections in the RAW, RECO and AOD data tiers;
regenerate the python dumps.

fwyzard · 2017-11-14T10:37:39Z

@cmsbuild, please test

cmsbuild · 2017-11-14T10:37:51Z

The code-checks are being triggered in jenkins.

cmsbuild · 2017-11-14T10:39:49Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21297/1984

cmsbuild · 2017-11-14T10:40:08Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24415/console Started: 2017/11/14 11:40

cmsbuild · 2017-11-14T10:40:10Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for master.

It involves the following packages:

HLTrigger/Configuration

@Martin-Grunewald, @silviodonato, @cmsbuild, @fwyzard can you please review it and eventually sign? Thanks.
@makortel, @Martin-Grunewald, @geoff-smith, @jalimena this is something you requested to watch as well.
@davidlange6, @slava77 you are the release manager for this.

cms-bot commands are listed here

cmsbuild · 2017-11-14T12:55:12Z

-1

Tested at: 6e4bd8d

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
45cd82f
2a6f759
b179c6e
23bbcdb
695af7d
54e6fc5
2b7bef5
59bf07c
71c7030
3139292
1b987aa
68359fa
5686e5c
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24415/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24415/git-merge-result

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24415/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

RelVals:

When I ran the RelVals I found an error in the following worklfows:
7.3 step2

runTheMatrix-results/7.3_CosmicsSPLoose_UP17+CosmicsSPLoose_UP17+DIGICOS_UP17+RECOCOS_UP17+ALCACOS_UP17+HARVESTCOS_UP17/step2_CosmicsSPLoose_UP17+CosmicsSPLoose_UP17+DIGICOS_UP17+RECOCOS_UP17+ALCACOS_UP17+HARVESTCOS_UP17.log

136.788 step2

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step2_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

10042.0 step2

runTheMatrix-results/10042.0_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step2_ZMM_13+ZMM_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

10024.0 step2

runTheMatrix-results/10024.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017/step2_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017_GenSimFull+DigiFull_2017+RecoFull_2017+ALCAFull_2017+HARVESTFull_2017.log

10224.0 step2

runTheMatrix-results/10224.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU/step2_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2017PU_GenSimFull+DigiFullPU_2017PU+RecoFullPU_2017PU+HARVESTFullPU_2017PU.log

11624.0 step2

runTheMatrix-results/11624.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2019_GenSimFull+DigiFull_2019+RecoFull_2019+ALCAFull_2019+HARVESTFull_2019/step2_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2019_GenSimFull+DigiFull_2019+RecoFull_2019+ALCAFull_2019+HARVESTFull_2019.log

10824.0 step2

runTheMatrix-results/10824.0_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018/step2_TTbar_13+TTbar_13TeV_TuneCUETP8M1_2018_GenSimFull+DigiFull_2018+RecoFull_2018+ALCAFull_2018+HARVESTFull_2018.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s HLT:2e34v31,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_2e34v31 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v31_MC.root --fileout file:RelVal_Raw_2e34v31_MC_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:27:38 2017-date Tue Nov 14 13:17:29 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:22:55 2017-date Tue Nov 14 13:17:39 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:28:10 2017-date Tue Nov 14 13:17:44 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:2e34v40,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_2e34v40 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v40_MC.root --fileout file:RelVal_Raw_2e34v40_MC_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:34:49 2017-date Tue Nov 14 13:22:53 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:2e34v31,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_2e34v31 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v31_DATA.root --fileout file:RelVal_Raw_2e34v31_DATA_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:30:07 2017-date Tue Nov 14 13:23:04 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:2e34v22,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_2e34v22 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v22_MC.root --fileout file:RelVal_Raw_2e34v22_MC_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:38:31 2017-date Tue Nov 14 13:27:45 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:2e34v22,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_2e34v22 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v22_DATA.root --fileout file:RelVal_Raw_2e34v22_DATA_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:34:02 2017-date Tue Nov 14 13:28:20 2017 s - exit: 256
cmsDriver.py RelVal -s HLT:2e34v40,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_2e34v40 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2017 --processName=HLTRECO --filein file:RelVal_Raw_2e34v40_DATA.root --fileout file:RelVal_Raw_2e34v40_DATA_HLT_RECO.root : FAILED - time: date Tue Nov 14 13:42:03 2017-date Tue Nov 14 13:34:53 2017 s - exit: 256

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
45cd82f
2a6f759
b179c6e
23bbcdb
695af7d
54e6fc5
2b7bef5
59bf07c
71c7030
3139292
1b987aa
68359fa
5686e5c
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24415/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24415/git-merge-result

cmsbuild · 2017-11-14T12:55:14Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

- keep DQMHistograms - keep Scouting* producers, drop the Scouting OutputModules, and change the Scouting EndPaths into regular Paths

cmsbuild · 2017-11-14T14:20:11Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/PR-21297/1987

cmsbuild · 2017-11-14T14:20:30Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/24420/console Started: 2017/11/14 15:20

cmsbuild · 2017-11-14T14:20:31Z

Pull request #21297 was updated. @Martin-Grunewald, @silviodonato, @cmsbuild, @fwyzard can you please check and sign again.

fwyzard · 2017-11-14T14:20:39Z

+1

cmsbuild · 2017-11-14T14:20:51Z

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar (and backports should be raised in the release meeting by the corresponding L2)

cmsbuild · 2017-11-14T16:56:25Z

+1
Tested at: 7e0d070
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24420/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:
21eca4e
14a581b
499f659
45cd82f
You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24420/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24420/git-merge-result

cmsbuild · 2017-11-14T16:56:28Z

Comparison job queued.

cmsbuild · 2017-11-14T18:46:33Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-21297/24420/summary.html

There are some workflows for which there are errors in the baseline:
10824.0 step 5
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Comparison Summary:

No significant changes to the logs found
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 27
DQMHistoTests: Total histograms compared: 2716578
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 7
DQMHistoTests: Total successes: 2716394
DQMHistoTests: Total skipped: 176
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 1.20000000016 KiB( 23 files compared)
Checked 111 log files, 8 edm output root files, 27 DQM output files

davidlange6 · 2017-11-14T19:46:12Z

HLTrigger/Configuration/python/HLTrigger_EventContent_cff.py

+        'keep *_hltScoutingPFPacker_*_*',
+        'keep *_hltScoutingPrimaryVertexPackerCaloMuon_*_*',
+        'keep *_hltScoutingPrimaryVertexPacker_*_*',
+        'keep edmTriggerResults_*_*_*'


hi @fwyzard - i guess we don't need this keep statement twice?

if you mean just

'keep edmTriggerResults_*_*_*'

no, we don't.

It is there because I've merged the keep statements from two different output modules, and this was the easiest way to do it.

I can try to find a better way...

davidlange6 · 2017-11-14T19:48:44Z

this adds

Product added ScoutingParticles_hltScoutingPFPacker__HLT.
Product added FEDRawDataCollection_hltFEDSelectorL1__HLT.
Product added ScoutingPFJets_hltScoutingPFPacker__HLT.
Product added ScoutingVertexs_hltScoutingPrimaryVertexPacker_primaryVtx_HLT.
Product added ScoutingCaloJets_hltScoutingCaloPacker__HLT.
Product added ScoutingVertexs_hltScoutingPrimaryVertexPackerCaloMuon_primaryVtx_HLT.
Product added ScoutingElectrons_hltScoutingEgammaPacker__HLT.
Product added ScoutingMuons_hltScoutingMuonPackerCalo__HLT.
Product added ScoutingPhotons_hltScoutingEgammaPacker__HLT.
Product added ScoutingVertexs_hltScoutingMuonPackerCalo_displacedVtx_HLT.
Product added ScoutingVertexs_hltScoutingMuonPacker_displacedVtx_HLT.
Product added ScoutingMuons_hltScoutingMuonPacker__HLT.
Product added double_hltScoutingCaloPacker_caloMetPhi_HLT.
Product added double_hltScoutingCaloPacker_caloMetPt_HLT.
Product added double_hltScoutingPFPacker_pfMetPhi_HLT.
Product added double_hltScoutingPFPacker_pfMetPt_HLT.
Product added double_hltScoutingCaloPacker_rho_HLT.
Product added double_hltScoutingPFPacker_rho_HLT.

davidlange6 · 2017-11-14T19:48:51Z

+1

cmsbuild added this to the CMSSW_10_0_X milestone Nov 14, 2017

fwyzard mentioned this pull request Nov 14, 2017

Include "Scouting" packers and collections in MC production #21295

Merged

cmsbuild added code-checks-pending comparison-pending hlt-pending orp-pending pending-signatures tests-pending labels Nov 14, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 14, 2017

cmsbuild added tests-started and removed tests-pending labels Nov 14, 2017

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Nov 14, 2017

fwyzard added 4 commits November 14, 2017 14:52

Update EndPaths kept in a cff python dump

ea7c1e6

- keep DQMHistograms - keep Scouting* producers, drop the Scouting OutputModules, and change the Scouting EndPaths into regular Paths

Keep the Scouting collections in the RAW, RECO and AOD data tiers

e27941c

Drop obsolete options --online/--offline

8499239

Regenerate python dumps to keep the Scouting collections

7e0d070

fwyzard force-pushed the update_Scouting_in_python_dumps_10x branch from 6e4bd8d to 7e0d070 Compare November 14, 2017 14:17

cmsbuild removed code-checks-approved comparison-notrun labels Nov 14, 2017

cmsbuild added code-checks-approved and removed code-checks-pending labels Nov 14, 2017

cmsbuild added fully-signed hlt-approved tests-started and removed hlt-pending pending-signatures tests-pending labels Nov 14, 2017

cmsbuild added tests-approved and removed tests-started labels Nov 14, 2017

cmsbuild added comparison-available and removed comparison-pending labels Nov 14, 2017

davidlange6 reviewed Nov 14, 2017

View reviewed changes

cmsbuild added orp-approved and removed orp-pending labels Nov 14, 2017

cmsbuild merged commit 43cb69e into cms-sw:master Nov 14, 2017

fwyzard deleted the update_Scouting_in_python_dumps_10x branch November 23, 2017 08:17

silviodonato mentioned this pull request Apr 12, 2018

Small bug fix in hltGetConfiguration (102X) #22943

Merged

This was referenced Feb 3, 2023

HLT scouting event content and producers [80X] #40691

Closed

include Scouting packers and collections in MC production [8_0_X] #40708

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Include "Scouting" packers and collections in MC production #21297

Include "Scouting" packers and collections in MC production #21297

fwyzard commented Nov 14, 2017

fwyzard commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017 •

edited

Loading

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017 •

edited

Loading

cmsbuild commented Nov 14, 2017

fwyzard commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

davidlange6 Nov 14, 2017

fwyzard Nov 14, 2017

davidlange6 commented Nov 14, 2017

davidlange6 commented Nov 14, 2017

Include "Scouting" packers and collections in MC production #21297

Include "Scouting" packers and collections in MC production #21297

Conversation

fwyzard commented Nov 14, 2017

fwyzard commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017 • edited Loading

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017 • edited Loading

cmsbuild commented Nov 14, 2017

fwyzard commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

cmsbuild commented Nov 14, 2017

davidlange6 Nov 14, 2017

Choose a reason for hiding this comment

fwyzard Nov 14, 2017

Choose a reason for hiding this comment

davidlange6 commented Nov 14, 2017

davidlange6 commented Nov 14, 2017

cmsbuild commented Nov 14, 2017 •

edited

Loading

cmsbuild commented Nov 14, 2017 •

edited

Loading