LowPtElectrons: convert Seed BDTs from XML to ROOT file format #22

bainbrid · 2021-08-17T13:23:05Z

This PR converts the formats (from XML to ROOT) of two files containing the weights of two BDT models used by the low pT electron seeding step in the reconstruction chain.

The effect is to reduce the file size by a factor two for both files, and to reduce the memory consumption and CPU time when parsing the weights files.

This was done in response to the issues raised here: cms-sw/cmssw#34707

A similar change was made previously, but on the weights file for the BDT model used by the low pT electron ID module (as opposed to the seeding step, as above), which solved the issues raised here: cms-sw/cmssw#28780

cmsbuild · 2021-08-17T13:23:26Z

A new Pull Request was created by @bainbrid for branch master.

@perrotta, @smuzaffar, @iarspider, @mrodozov, @cmsbuild, @slava77, @jpata can you please review it and eventually sign? Thanks.
@Sam-Harper, @jainshilpi, @lgray, @sobhatta, @afiqaize, @varuns23, @ram1123 this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.
cms-bot commands are listed here

bainbrid · 2021-08-17T13:24:00Z

This PR is used by cms-sw/cmssw#34908

slava77 · 2021-08-19T13:55:01Z

+reconstruction

for cmsdist this should be merged together with cms-sw/cmssw#34908

perrotta · 2021-08-19T15:03:47Z

This has to be merged together with #34908: @smuzaffar (just to learn how to deal with it in the future) how do we manage to synchronize the merging of the PR in cms-data and in cmssw? I think I should wait for a "+1" from externals, once tests finish: then we'll merge the two PRs at the same time, so that they end up in the same IB?
By the way, this cms-data PR was already succesfully tested together with #34908, while now it reports about an error from a addOn test: let see what is it...

smuzaffar · 2021-08-19T15:16:06Z

+externals
@perrotta , once cms-data external PR is ready (means already successfully tested directly via cms-data PR or indirectly via cmssw PR) then we merge it and bot automatically tags the cms-data changes and creates a cmsdist PR. If you want then you can restart a fresh test to test cmsdist cmssw PRs otheriwse normally it is ok to just merge the cmsdist and cmssw PRs.

Because of +reco comment bot started the test here and as it was ran without cmssw PR that is why there are failures. I would suggest to ignore these errors and goahead with the merge. Testing it via cmssw was good enough.

cmsbuild · 2021-08-19T15:16:30Z

This pull request is fully signed and it will be integrated in one of the next master IBs after it passes the integration tests. This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

perrotta · 2021-08-19T15:23:57Z

+1

To go together with #34908

perrotta · 2021-08-19T15:25:07Z

merge

cmsbuild · 2021-08-19T15:33:07Z

-1

Failed Tests: RelVals RelVals-INPUT AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b45164/17891/summary.html
COMMIT: 8e4d325
CMSSW: CMSSW_12_1_X_2021-08-19-1100/slc7_amd64_gcc900
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/install.sh to create a dev area with all the needed externals and cmssw changes.

RelVals

135.4

----- Begin Fatal Exception 19-Aug-2021 16:28:17 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeedsTmp'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/135.4_ZEE_13+ZEEFS_13+HARVESTUP15FS+MINIAODMCUP15FS
----- End Fatal Exception -------------------------------------------------

5.1

----- Begin Fatal Exception 19-Aug-2021 16:28:17 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeedsTmp'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/5.1_TTbar+TTbarFS+HARVESTFS
----- End Fatal Exception -------------------------------------------------

140.56

----- Begin Fatal Exception 19-Aug-2021 16:30:01 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeeds'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-relvals/runTheMatrix-results/140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18
----- End Fatal Exception -------------------------------------------------

Expand to see more relval errors ...

RelVals-INPUT

4.64.6_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1/step2_MinimumBias2010A+MinimumBias2010A+RECOSKIMALCA+HARVESTDR1.log
140.54140.54_RunPA2013+RunPA2013+RECO_PPbData+HARVEST_PPbData/step2_RunPA2013+RunPA2013+RECO_PPbData+HARVEST_PPbData.log
140.56140.56_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18/step2_RunHI2018+RunHI2018+RECOHID18+HARVESTDHI18.log

Expand to see more relval errors ...

AddOn Tests

fastsim

----- Begin Fatal Exception 19-Aug-2021 16:27:55 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeedsTmp'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/fastsim
----- End Fatal Exception -------------------------------------------------

fastsim1

----- Begin Fatal Exception 19-Aug-2021 16:27:55 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeedsTmp'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/fastsim1
----- End Fatal Exception -------------------------------------------------

fastsim2

----- Begin Fatal Exception 19-Aug-2021 16:27:57 CEST-----------------------
An exception of category 'FileInPathError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing module: class=LowPtGsfElectronSeedProducer label='lowPtGsfElectronSeedsTmp'
Exception Message:
edm::FileInPath unable to find file RecoEgamma/ElectronIdentification/data/LowPtElectrons/RunII_Autumn18_LowPtElectrons_unbiased.xml.gz anywhere in the search path.
The search path is defined by: CMSSW_SEARCH_PATH
${CMSSW_SEARCH_PATH} is: /cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/src:/cvmfs/cms-ci.cern.ch/week0/cms-data/RecoEgamma-ElectronIdentification/22/17891/CMSSW_12_1_X_2021-08-19-1100/external/slc7_amd64_gcc900/data:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/poison:/cvmfs/cms-ib.cern.ch/week0/slc7_amd64_gcc900/cms/cmssw-patch/CMSSW_12_1_X_2021-08-19-1100/src
Current directory is: /data/cmsbld/jenkins/workspace/ib-run-pr-addon/addOnTests/fastsim2
----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

bainbrid · 2021-08-19T18:19:54Z

Is this #22 (comment) something I need to worry about? Presumably so?

Why is it only showing up now?

mrodozov · 2021-08-19T18:26:29Z

because the test has been started, and when the pr was merged the test wasn't stopped. and also because it ran without the corresponding cmssw PR

convert Seed BDTs from XML to ROOT file format

8e4d325

cmsbuild added externals-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Aug 17, 2021

bainbrid mentioned this pull request Aug 17, 2021

LowPtElectrons: convert Seed BDTs from XML to ROOT file format cms-sw/cmssw#34908

Merged

cmsbuild added reconstruction-approved tests-started and removed tests-pending reconstruction-pending labels Aug 19, 2021

cmsbuild added externals-approved fully-signed and removed pending-signatures externals-pending labels Aug 19, 2021

cmsbuild added orp-approved and removed orp-pending labels Aug 19, 2021

cmsbuild merged commit 977b378 into cms-data:master Aug 19, 2021

cmsbuild mentioned this pull request Aug 19, 2021

Update tag for RecoEgamma-ElectronIdentification to V01-07-00 cms-sw/cmsdist#7220

Merged

cmsbuild added tests-rejected and removed tests-started labels Aug 19, 2021

gartung mentioned this pull request Aug 19, 2021

Add generation of igprof memory profile report for pull requests cms-sw/cms-bot#1610

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

LowPtElectrons: convert Seed BDTs from XML to ROOT file format #22

LowPtElectrons: convert Seed BDTs from XML to ROOT file format #22

bainbrid commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

bainbrid commented Aug 17, 2021

slava77 commented Aug 19, 2021

perrotta commented Aug 19, 2021

smuzaffar commented Aug 19, 2021

cmsbuild commented Aug 19, 2021

perrotta commented Aug 19, 2021

perrotta commented Aug 19, 2021

cmsbuild commented Aug 19, 2021

bainbrid commented Aug 19, 2021

mrodozov commented Aug 19, 2021 •

edited

LowPtElectrons: convert Seed BDTs from XML to ROOT file format #22

LowPtElectrons: convert Seed BDTs from XML to ROOT file format #22

Conversation

bainbrid commented Aug 17, 2021

cmsbuild commented Aug 17, 2021

bainbrid commented Aug 17, 2021

slava77 commented Aug 19, 2021

perrotta commented Aug 19, 2021

smuzaffar commented Aug 19, 2021

cmsbuild commented Aug 19, 2021

perrotta commented Aug 19, 2021

perrotta commented Aug 19, 2021

cmsbuild commented Aug 19, 2021

RelVals

RelVals-INPUT

AddOn Tests

bainbrid commented Aug 19, 2021

mrodozov commented Aug 19, 2021 • edited

mrodozov commented Aug 19, 2021 •

edited