Reconfigure particle flow config files #28110

hatakeyamak · 2019-10-03T14:22:34Z

PR description:

Reconfigure particleFlow_cfi and make clear the parameters used for PFEGammaFilters, PFMuonAlgo, i.e. different components of PFAlgo. Also used this opportunity to remove obsolete config parameters.

This is a follow-up of
#27782
and fillDescriptions are introduced, and reduced ParameterSets use a little bit wrt PR27782

PR validation:

Compiles and run a few matrix tests with success (scenarios for 2017, 2018, and phase2)

if this PR is a backport please specify the original PR:

This is not a backport PR.

@bendavid @jpata @guitargeek

cmsbuild · 2019-10-03T14:23:05Z

The code-checks are being triggered in jenkins.

hatakeyamak · 2019-10-03T14:23:55Z

Actually, this will fail the jenkins tests that runs HLT. I am not familiar with what to do to update HLTrigger/Configuration/python/HLT_GRun_cff.py reflecting the changes here, and suggestions will be appreciated.

cmsbuild · 2019-10-03T14:29:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-28110/12125

This PR adds an extra 208KB to repository

cmsbuild · 2019-10-03T14:29:34Z

A new Pull Request was created by @hatakeyamak (Kenichi Hatakeyama) for master.

It involves the following packages:

RecoParticleFlow/PFProducer

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@mmarionncern, @lgray, @seemasharmafnal, @bachtis, @cbernet this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

slava77 · 2019-10-03T14:30:28Z

@cmsbuild please test

cmsbuild · 2019-10-03T14:30:53Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/2776/console Started: 2019/10/03 16:31

Martin-Grunewald · 2019-10-03T14:50:38Z

RecoParticleFlow/PFProducer/plugins/importers/GeneralTracksImporter.cc

-    pfmu_ = std::unique_ptr<PFMuonAlgo>(new PFMuonAlgo(conf));
+    bool postMuonCleaning =
+        conf.existsAs<bool>("postMuonCleaning") ? conf.getParameter<bool>("postMuonCleaning") : false;
+    pfmu_ = std::unique_ptr<PFMuonAlgo>(new PFMuonAlgo(conf, postMuonCleaning));


Why is this existsAs needed here (and in some other locations) if the parameter is part of fillDescriptions ayway?

Thanks for taking a look. This comes from PFBlockProducer.cc (not PFProducer.cc) for which we haven't introduced fillDescriptions yet...

cmsbuild · 2019-10-03T15:41:18Z

-1

Tested at: 5ea0bf6

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f6d58/2776/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

RelVals:

When I ran the RelVals I found an error in the following workflows:
140.56 step2

runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

158.0 step2

runTheMatrix-results/158.0_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO/step2_HydjetQ_B12_5020GeV_2018_ppReco+HydjetQ_B12_5020GeV_2018_ppReco+DIGIHI2018PPRECO+RECOHI2018PPRECO+ALCARECOHI2018PPRECO+HARVESTHI2018PPRECO.log

11634.0 step2

runTheMatrix-results/11634.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021/step2_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2021_GenSimFull+DigiFull_2021+RecoFull_2021+HARVESTFull_2021+ALCAFull_2021.log

12434.0 step2

runTheMatrix-results/12434.0_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023/step2_TTbar_14TeV+TTbar_14TeV_TuneCUETP8M1_2023_GenSimFull+DigiFull_2023+RecoFull_2023+HARVESTFull_2023+ALCAFull_2023.log

AddOn:

I found errors in the following addon tests:

cmsRun /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Thu Oct 3 17:24:35 2019-date Thu Oct 3 17:23:23 2019 s - exit: 18688
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Thu Oct 3 17:24:35 2019-date Thu Oct 3 17:23:23 2019 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Thu Oct 3 17:30:20 2019-date Thu Oct 3 17:23:26 2019 s - exit: 18688
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run3 --processName=HLTRECO --filein file:RelVal_Raw_GRun_MC.root --fileout file:RelVal_Raw_GRun_MC_HLT_RECO.root : FAILED - time: date Thu Oct 3 17:30:20 2019-date Thu Oct 3 17:23:26 2019 s - exit: 18688
cmsRun /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=False globalTag=@ inputFiles=@ : FAILED - time: date Thu Oct 3 17:29:17 2019-date Thu Oct 3 17:23:48 2019 s - exit: 18688
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --mc --scenario=pp -n 10 --conditions auto:run2_mc_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018_pp_on_AA --processName=HLTRECO --filein file:RelVal_Raw_HIon_MC.root --fileout file:RelVal_Raw_HIon_MC_HLT_RECO.root : FAILED - time: date Thu Oct 3 17:29:17 2019-date Thu Oct 3 17:23:48 2019 s - exit: 256
cmsRun /cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc820/cms/cmssw-patch/CMSSW_11_0_X_2019-10-02-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Thu Oct 3 17:29:24 2019-date Thu Oct 3 17:24:02 2019 s - exit: 18688
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Thu Oct 3 17:29:24 2019-date Thu Oct 3 17:24:02 2019 s - exit: 18688

cmsbuild · 2019-10-03T15:41:21Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

cmsbuild · 2019-10-30T13:29:44Z

Pull request #28110 was updated. @perrotta, @Martin-Grunewald, @cmsbuild, @fwyzard, @slava77 can you please check and sign again.

hatakeyamak · 2019-10-30T13:32:03Z

Thank you @fabiocos . I think I solved the conflict.
Does this PR look good now, reviewers?

slava77 · 2019-10-30T14:06:07Z

@cmsbuild please test

cmsbuild · 2019-10-30T14:06:37Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/3254/console Started: 2019/10/30 15:07

cmsbuild · 2019-10-30T19:34:07Z

+1
Tested at: 6756673
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f6d58/3254/summary.html

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

call close with the option to shutdown workers #28305 @dan131riley: call close with the option to shutdown workers
Remove ParameterSet::Bool as unused #28294 @makortel: Remove ParameterSet::Bool as unused
Use corrected PF cluster correction tags for 2018 design MC scenario #28263 @christopheralanwest: Use corrected PF cluster correction tags for 2018 design MC scenario

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f6d58/3254/git-log-recent-commits
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f6d58/3254/git-merge-result

cmsbuild · 2019-10-30T19:34:12Z

Comparison job queued.

cmsbuild · 2019-10-30T21:29:13Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-2f6d58/3254/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 34
DQMHistoTests: Total histograms compared: 2961036
DQMHistoTests: Total failures: 1
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 2960694
DQMHistoTests: Total skipped: 341
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 33 files compared)
Checked 147 log files, 16 edm output root files, 34 DQM output files

slava77 · 2019-11-01T13:31:13Z

+1

for #28110 6756673

code changes are in line with the PR description and the follow up review
- I only inspected the value changes from the code diffs, additionally relying on lack of differences in the results
jenkins tests pass and comparisons with the baseline show no differences

hatakeyamak · 2019-11-04T10:41:23Z

Thanks @slava77.
Looks like this is still waiting for HLT signature. @fwyzard @Martin-Grunewald

Martin-Grunewald · 2019-11-04T13:04:08Z

+1

cmsbuild · 2019-11-04T13:04:37Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2019-11-04T20:57:40Z

+1

hatakeyamak added 10 commits August 14, 2019 14:42

Clean up the particleFlow config file and make associated changes

3712eb3

code-check

6e482a4

code-format

ccd41d9

Clean up the particleFlow config file and make associated changes

cdc2c1a

code-check

01af87e

code-format

b33229f

introduce fillDescriptions

f3c8a52

solve conflicts

2e5bb5f

solve conflicts

1526106

cleanup. reduce parametersets usage somewhat.

5ea0bf6

cmsbuild added this to the CMSSW_11_0_X milestone Oct 3, 2019

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 3, 2019

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 3, 2019

cmsbuild added tests-started and removed tests-pending labels Oct 3, 2019

Martin-Grunewald reviewed Oct 3, 2019

View reviewed changes

cmsbuild added the code-checks-approved label Oct 30, 2019

cmsbuild added tests-started and removed tests-pending labels Oct 30, 2019

cmsbuild added tests-approved and removed tests-started labels Oct 30, 2019

cmsbuild added comparison-available and removed comparison-pending labels Oct 30, 2019

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Nov 1, 2019

cmsbuild added fully-signed hlt-approved and removed hlt-pending pending-signatures labels Nov 4, 2019

cmsbuild added orp-approved and removed orp-pending labels Nov 4, 2019

cmsbuild merged commit 5762ef9 into cms-sw:master Nov 4, 2019

cmsbuild mentioned this pull request Nov 5, 2019

Apply code formatting to FWCore/Framework #28345

Merged

Sam-Harper mentioned this pull request Nov 15, 2019

Broken HLT customisation function #28403

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reconfigure particle flow config files #28110

Reconfigure particle flow config files #28110

hatakeyamak commented Oct 3, 2019 •

edited

cmsbuild commented Oct 3, 2019

hatakeyamak commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

slava77 commented Oct 3, 2019

cmsbuild commented Oct 3, 2019 •

edited

Martin-Grunewald Oct 3, 2019

hatakeyamak Oct 3, 2019 •

edited

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 30, 2019

hatakeyamak commented Oct 30, 2019

slava77 commented Oct 30, 2019

cmsbuild commented Oct 30, 2019 •

edited

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

slava77 commented Nov 1, 2019

hatakeyamak commented Nov 4, 2019

Martin-Grunewald commented Nov 4, 2019

cmsbuild commented Nov 4, 2019

fabiocos commented Nov 4, 2019

Reconfigure particle flow config files #28110

Reconfigure particle flow config files #28110

Conversation

hatakeyamak commented Oct 3, 2019 • edited

PR description:

PR validation:

if this PR is a backport please specify the original PR:

cmsbuild commented Oct 3, 2019

hatakeyamak commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

slava77 commented Oct 3, 2019

cmsbuild commented Oct 3, 2019 • edited

Martin-Grunewald Oct 3, 2019

Choose a reason for hiding this comment

hatakeyamak Oct 3, 2019 • edited

Choose a reason for hiding this comment

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 3, 2019

cmsbuild commented Oct 30, 2019

hatakeyamak commented Oct 30, 2019

slava77 commented Oct 30, 2019

cmsbuild commented Oct 30, 2019 • edited

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

cmsbuild commented Oct 30, 2019

slava77 commented Nov 1, 2019

hatakeyamak commented Nov 4, 2019

Martin-Grunewald commented Nov 4, 2019

cmsbuild commented Nov 4, 2019

fabiocos commented Nov 4, 2019

hatakeyamak commented Oct 3, 2019 •

edited

cmsbuild commented Oct 3, 2019 •

edited

hatakeyamak Oct 3, 2019 •

edited

cmsbuild commented Oct 30, 2019 •

edited