Eliminate an infinite loop from MVAVariableManager<T>::init() #24756

makortel · 2018-09-28T14:19:07Z

While doing something unrelated, git+scram got into a state thinking that the photon ID MVA text files are removed, and thus the MVAVariableManager<T>::init() got a file with only content of #error ... that lead to an infinite loop. This PR suggests to add a check for the EOF also when reading the name, eliminating the infinite loop (and leading to a later exception in my test case).

Tested in CMSSW_10_3_X_2018-09-26-1100, no changes expected.

cmsbuild · 2018-09-28T14:19:28Z

The code-checks are being triggered in jenkins.

cmsbuild · 2018-09-28T14:23:03Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-24756/6699

cmsbuild · 2018-09-28T14:23:22Z

A new Pull Request was created by @makortel (Matti Kortelainen) for master.

It involves the following packages:

RecoEgamma/EgammaTools

@perrotta, @cmsbuild, @slava77 can you please review it and eventually sign? Thanks.
@jainshilpi, @Sam-Harper, @varuns23, @lgray this is something you requested to watch as well.
@davidlange6, @slava77, @fabiocos you are the release manager for this.

cms-bot commands are listed here

makortel · 2018-09-28T14:24:24Z

@cmsbuild, please test

cmsbuild · 2018-09-28T14:24:42Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30771/console Started: 2018/09/28 17:37

cmsbuild · 2018-09-28T18:03:05Z

-1

Tested at: 9c0345a

You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24756/30771/summary.html

I found follow errors while testing this PR

Failed tests: RelVals AddOn

RelVals:

When I ran the RelVals I found an error in the following worklfows:
136.731 step2

runTheMatrix-results/136.731_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2/step2_RunSinglePh2016B+RunSinglePh2016B+HLTDR2_2016+RECODR2_2016reHLT_skimSinglePh_HIPM+HARVESTDR2.log

136.788 step2

runTheMatrix-results/136.788_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017/step2_RunSinglePh2017B+RunSinglePh2017B+HLTDR2_2017+RECODR2_2017reHLT_skimSinglePh_Prompt+HARVEST2017.log

1325.7 step2

runTheMatrix-results/1325.7_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2/step2_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2.log

1000.0 step2

runTheMatrix-results/1000.0_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT/step2_RunMinBias2011A+RunMinBias2011A+TIER0+SKIMD+HARVESTDfst2+ALCASPLIT.log

1001.0 step2

runTheMatrix-results/1001.0_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5/step2_RunMinBias2011A+RunMinBias2011A+TIER0EXP+ALCAEXP+ALCAHARVDSIPIXELCALRUN1+ALCAHARVD1+ALCAHARVD2+ALCAHARVD3+ALCAHARVD4+ALCAHARVD5.log

AddOn:

I found errors in the following addon tests:

cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_Fake2 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016 --fileout file:RelVal_Raw_Fake2_DATA.root --filein /store/data/Run2016B/JetHT/RAW/v1/000/272/762/00000/C666CDE2-E013-E611-B15A-02163E011DBE.root : FAILED - time: date Fri Sep 28 19:34:42 2018-date Fri Sep 28 19:33:59 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake2.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:34:42 2018-date Fri Sep 28 19:33:59 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake2,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake2 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016 --processName=HLTRECO --filein file:RelVal_Raw_Fake2_DATA.root --fileout file:RelVal_Raw_Fake2_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:34:42 2018-date Fri Sep 28 19:33:59 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PRef --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --eventcontent RAW --fileout file:RelVal_Raw_PRef_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Fri Sep 28 19:34:55 2018-date Fri Sep 28 19:34:03 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PRef.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:34:55 2018-date Fri Sep 28 19:34:03 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:PRef,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PRef --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PRef_DATA.root --fileout file:RelVal_Raw_PRef_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:34:55 2018-date Fri Sep 28 19:34:03 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full2015Data --data --scenario=HeavyIons -n 10 --conditions auto:run2_hlt_HIon --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2016,Run2_HI --fileout file:RelVal_Raw_HIon_DATA.root --filein /store/hidata/HIRun2015/HIHardProbes/RAW-RECO/HighPtJet-PromptReco-v1/000/263/689/00000/1802CD9A-DDB8-E511-9CF9-02163E0138CA.root : FAILED - time: date Fri Sep 28 19:34:59 2018-date Fri Sep 28 19:34:04 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_HIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:34:59 2018-date Fri Sep 28 19:34:04 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:HIon,RAW2DIGI,L1Reco,RECO --data --scenario=HeavyIons -n 10 --conditions auto:run2_data_HIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2016,Run2_HI --processName=HLTRECO --filein file:RelVal_Raw_HIon_DATA.root --fileout file:RelVal_Raw_HIon_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:34:59 2018-date Fri Sep 28 19:34:04 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_PIon --relval 9000,50 --datatier "RAW" --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --eventcontent RAW --fileout file:RelVal_Raw_PIon_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Fri Sep 28 19:38:55 2018-date Fri Sep 28 19:35:01 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_PIon.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:38:55 2018-date Fri Sep 28 19:35:01 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:PIon,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_PIon --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_PIon_DATA.root --fileout file:RelVal_Raw_PIon_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:38:55 2018-date Fri Sep 28 19:35:01 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_2018v35 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_2018v35_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Fri Sep 28 19:45:07 2018-date Fri Sep 28 19:38:59 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_2018v35.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:45:07 2018-date Fri Sep 28 19:38:59 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:2018v35,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_2018v35 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_2018v35_DATA.root --fileout file:RelVal_Raw_2018v35_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:45:07 2018-date Fri Sep 28 19:38:59 2018 s - exit: 21504
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/Utilities/ReleaseScripts/scripts/read312RV_cfg.py : FAILED - time: date Fri Sep 28 19:44:36 2018-date Fri Sep 28 19:44:17 2018 s - exit: 23552
cmsDriver.py RelVal -s L1REPACK:GT1 --data --scenario=pp -n 10 --conditions auto:run1_hlt_Fake --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --fileout file:RelVal_Raw_Fake_DATA.root --filein /store/data/Run2012A/MuEG/RAW/v1/000/191/718/14932935-E289-E111-830C-5404A6388697.root : FAILED - time: date Fri Sep 28 19:46:39 2018-date Fri Sep 28 19:44:39 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:46:39 2018-date Fri Sep 28 19:44:39 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run1_data_Fake --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --processName=HLTRECO --filein file:RelVal_Raw_Fake_DATA.root --fileout file:RelVal_Raw_Fake_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:46:39 2018-date Fri Sep 28 19:44:39 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:Full --data --scenario=pp -n 10 --conditions auto:run2_hlt_GRun --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_2018 --fileout file:RelVal_Raw_GRun_DATA.root --filein /store/data/Run2017A/HLTPhysics4/RAW/v1/000/295/606/00000/36DE5E0A-3645-E711-8FA1-02163E01A43B.root : FAILED - time: date Fri Sep 28 19:47:38 2018-date Fri Sep 28 19:45:12 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_GRun.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:47:38 2018-date Fri Sep 28 19:45:12 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:GRun,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_GRun --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_2018 --processName=HLTRECO --filein file:RelVal_Raw_GRun_DATA.root --fileout file:RelVal_Raw_GRun_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:47:38 2018-date Fri Sep 28 19:45:12 2018 s - exit: 21504
cmsDriver.py RelVal -s L1REPACK:GCTGT --data --scenario=pp -n 10 --conditions auto:run2_hlt_Fake1 --relval 9000,50 --datatier "RAW" --eventcontent RAW --customise=HLTrigger/Configuration/CustomConfigs.L1T --era Run2_25ns --fileout file:RelVal_Raw_Fake1_DATA.root --filein /store/data/Run2015D/MuonEG/RAW/v1/000/256/677/00000/80950A90-745D-E511-92FD-02163E011C5D.root : FAILED - time: date Fri Sep 28 19:47:49 2018-date Fri Sep 28 19:46:46 2018 s - exit: 23552
cmsRun /cvmfs/cms-ib.cern.ch/nweek-02543/slc6_amd64_gcc700/cms/cmssw/CMSSW_10_3_X_2018-09-27-2300/src/HLTrigger/Configuration/test/OnLine_HLT_Fake1.py realData=True globalTag=@ inputFiles=@ : FAILED - time: date Fri Sep 28 19:47:49 2018-date Fri Sep 28 19:46:46 2018 s - exit: 21504
cmsDriver.py RelVal -s HLT:Fake1,RAW2DIGI,L1Reco,RECO --data --scenario=pp -n 10 --conditions auto:run2_data_Fake1 --relval 9000,50 --datatier "RAW-HLT-RECO" --eventcontent FEVTDEBUGHLT --customise=HLTrigger/Configuration/CustomConfigs.L1THLT --era Run2_25ns --processName=HLTRECO --filein file:RelVal_Raw_Fake1_DATA.root --fileout file:RelVal_Raw_Fake1_DATA_HLT_RECO.root : FAILED - time: date Fri Sep 28 19:47:49 2018-date Fri Sep 28 19:46:46 2018 s - exit: 21504

cmsbuild · 2018-09-28T18:03:07Z

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

makortel · 2018-09-28T18:08:37Z

Umm

Exception Message:
Failed to open the file 'root://cms-xrd-global.cern.ch//store/data/Run2016B/SinglePhoton/RAW/v2/000/274/199/00000/007498A6-8426-E611-8BA8-02163E013754.root'
   Additional Info:
      [a] XrdCl::File::Open(name='root://eoscms.cern.ch//eos/cms/store/user/cmsbuild/store/data/Run2016B/SinglePhoton/RAW/v2/000/274/199/00000/007498A6-8426-E611-8BA8-02163E013754.root', flags=0x10, permissions=0660) => error '[ERROR] Server responded with an error: [3010] Unable to give access - user access restricted - unauthorized identity used ; Permission denied

makortel · 2018-09-28T18:10:57Z

@cmsbuild, please test

Let's see if it was some transient error (all other failed workflows had similar error).

cmsbuild · 2018-09-28T18:11:16Z

The tests are being triggered in jenkins.
https://cmssdt.cern.ch/jenkins/job/ib-any-integration/30791/console Started: 2018/09/28 22:28

cmsbuild · 2018-09-28T23:00:27Z

+1
Tested at: 9c0345a
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24756/30791/summary.html

cmsbuild · 2018-09-28T23:00:30Z

Comparison job queued.

cmsbuild · 2018-09-29T02:54:28Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-24756/30791/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 32
DQMHistoTests: Total histograms compared: 3162160
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3161961
DQMHistoTests: Total skipped: 197
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 31 files compared)
Checked 134 log files, 14 edm output root files, 32 DQM output files

Sam-Harper · 2018-09-29T10:27:41Z

@makortel, thanks for fixing and sorry for the inconvenience caused

perrotta · 2018-10-02T09:03:14Z

+1

The PR does what is in the description. Infinite loops are avoided in some pathological sitiations
No changes are expected for "normal" input files:, and no changes are visible from jenkins tests

cmsbuild · 2018-10-02T09:03:34Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @davidlange6, @slava77, @smuzaffar, @fabiocos (and backports should be raised in the release meeting by the corresponding L2)

fabiocos · 2018-10-02T09:27:23Z

+1

Add check for eof() also for the name

9c0345a

cmsbuild added this to the CMSSW_10_3_X milestone Sep 28, 2018

cmsbuild added code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Sep 28, 2018

cmsbuild added code-checks-approved and removed code-checks-pending labels Sep 28, 2018

cmsbuild added tests-started and removed tests-pending labels Sep 28, 2018

cmsbuild added comparison-notrun tests-rejected and removed comparison-pending tests-started labels Sep 28, 2018

cmsbuild added comparison-pending tests-pending and removed comparison-notrun tests-rejected labels Sep 28, 2018

cmsbuild added tests-started and removed tests-pending labels Sep 28, 2018

cmsbuild added tests-approved and removed tests-started labels Sep 28, 2018

cmsbuild added comparison-available and removed comparison-pending labels Sep 29, 2018

cmsbuild added fully-signed reconstruction-approved and removed pending-signatures reconstruction-pending labels Oct 2, 2018

cmsbuild added orp-approved and removed orp-pending labels Oct 2, 2018

cmsbuild merged commit 31d8ec2 into cms-sw:master Oct 2, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Eliminate an infinite loop from MVAVariableManager<T>::init() #24756

Eliminate an infinite loop from MVAVariableManager<T>::init() #24756

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018 •

edited

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

makortel commented Sep 28, 2018

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018 •

edited

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 29, 2018

Sam-Harper commented Sep 29, 2018

perrotta commented Oct 2, 2018

cmsbuild commented Oct 2, 2018

fabiocos commented Oct 2, 2018

Eliminate an infinite loop from MVAVariableManager<T>::init() #24756

Eliminate an infinite loop from MVAVariableManager<T>::init() #24756

Conversation

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018 • edited

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

makortel commented Sep 28, 2018

makortel commented Sep 28, 2018

cmsbuild commented Sep 28, 2018 • edited

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 28, 2018

cmsbuild commented Sep 29, 2018

Sam-Harper commented Sep 29, 2018

perrotta commented Oct 2, 2018

cmsbuild commented Oct 2, 2018

fabiocos commented Oct 2, 2018

cmsbuild commented Sep 28, 2018 •

edited

cmsbuild commented Sep 28, 2018 •

edited