DeepVertex and DeepJet+DeepVertex combination in release (onnx inference) #31988

leonardogiannini · 2020-10-29T16:07:43Z

PR description:

The PR adds the new training of the DeepVertex taggers and the Combination with DeepJet. The BTV POG wants to have them into the release but not in the standard production for future deployment studies.

The BTV group will take care of enabling the taggers for a BTV only dedicated production.

In order to test, expand the configuration RecoBTag/ONNXRuntime/test/test_deep_vertexcomb_cfg.py
and turn to "True" the switches "compute_probabilities" and "run_deepVertex", which are set by default to "False" (module pfDeepFlavourTagInfos). Alternatively, modify RecoBTag/FeatureTools/plugins/DeepFlavourTagInfoProducer.cc before compiling.

PR validation:

the PR is presented here https://docs.google.com/presentation/d/1vWKUH2ANOMQWEdp-tT-9S8WW29RPKoe8m8zNzdk1O6M/edit#slide=id.ga4e5d7cc6d_0_79

After the meeting the agreement was to add taggers with ONNX inference

The performances validated in CMSSW are in slide 3 (purple line)

…pJet etc.

cmsbuild · 2020-10-29T16:08:08Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-29T16:15:49Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31988/19471

This PR adds an extra 24KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-format:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31988/19471/code-format.patch
e.g. curl https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31988/19471/code-format.patch | patch -p1
You can also run scram build code-format to apply code format directly

cmsbuild · 2020-10-29T16:50:41Z

The code-checks are being triggered in jenkins.

cmsbuild · 2020-10-29T16:57:51Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-31988/19472

This PR adds an extra 24KB to repository

cmsbuild · 2020-10-29T16:58:14Z

A new Pull Request was created by @leonardogiannini for master.

It involves the following packages:

PhysicsTools/PatAlgos
RecoBTag/ONNXRuntime

@perrotta, @jpata, @cmsbuild, @santocch, @slava77 can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @JyothsnaKomaragiri, @ahinzmann, @smoortga, @riga, @schoef, @mmarionncern, @jdamgov, @jdolen, @nhanvtran, @gkasieczka, @clelange, @emilbols, @hatakeyamak, @ferencek, @gpetruc, @andrzejnovak, @mariadalfonso, @seemasharmafnal this is something you requested to watch as well.
@silviodonato, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

santocch · 2020-11-04T10:44:10Z

please test

cmsbuild · 2020-11-04T10:44:34Z

The tests are being triggered in jenkins.

CMSSW_11_2_X_2020-11-03-2300/slc7_amd64_gcc820: https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/10494/console Started: 2020/11/04 11:46

cmsbuild · 2020-11-04T12:16:29Z

+1
Tested at: c01ee19
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-36ce35/10494/summary.html
CMSSW: CMSSW_11_2_X_2020-11-03-2300
SCRAM_ARCH: slc7_amd64_gcc820

cmsbuild · 2020-11-11T12:13:14Z

Comparison job queued.

leonardogiannini · 2020-11-11T12:53:16Z

The profiles for cpu and memory of a ReMiniAOD (step2 of 1325.518)
are here
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_mem
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_cpu

jpata · 2020-11-11T12:59:12Z

The profiles for cpu and memory of a ReMiniAOD (step2 of 1325.518)

Thanks for that, but I can't find DeepVertex in the latest profiles. Can you check if it was enabled in step2?

cmsbuild · 2020-11-11T13:30:32Z

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-36ce35/10636/summary.html

Comparison Summary:

No significant changes to the logs found
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 35
DQMHistoTests: Total histograms compared: 2529296
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 2529267
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: -0.004 KiB( 34 files compared)
DQMHistoSizes: changed ( 312.0 ): -0.004 KiB MessageLogger/Warnings
Checked 148 log files, 22 edm output root files, 35 DQM output files

leonardogiannini · 2020-11-11T13:48:42Z

The profiles for cpu and memory of a ReMiniAOD (step2 of 1325.518)

Thanks for that, but I can't find DeepVertex in the latest profiles. Can you check if it was enabled in step2?

I suppose it's below 1000. It appears in the timing reports as usual together with the other taggers

TimeReport 0.012164 0.012164 0.012164 pfDeepCombinedJetTagsSlimmedDeepFlavour
TimeReport 0.007313 0.007313 0.007313 pfDeepFlavourJetTagsSlimmedDeepFlavour
TimeReport 0.008506 0.008506 0.008506 pfDeepFlavourTagInfosSlimmedDeepFlavour
TimeReport 0.008442 0.008442 0.008442 pfDeepVertexJetTagsSlimmedDeepFlavour

Now the inference is bit more optimized wrt to the PU200 profile, as it runs only on interesting jets and there are less jets overall in these samples. (I am running on /store/relval/CMSSW_10_6_4/RelValProdTTbar_13_pmx25ns/AODSIM/PUpmx25ns_106X_upgrade2018_realistic_v9-v1/10000/*)

slava77 · 2020-11-11T14:44:55Z

The profiles for cpu and memory of a ReMiniAOD (step2 of 1325.518)
are here
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_mem
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_cpu

https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_cpu/17

%	from	total	c1	c tot
0.32	0.47	0.47	1	1	DeepCombinedONNXJetTagsProducer::produce(edm::Event&, edm::EventSetup const&)
0.19	0.27	0.27	1	1	DeepVertexONNXJetTagsProducer::produce(edm::Event&, edm::EventSetup const&)
0.17	0.25	0.25	1	1	DeepFlavourONNXJetTagsProducer::produce(edm::Event&, edm::EventSetup const&)

the "total" for the denominator is close to 35%.
So, if we run both DeepVertex and DeepCombined, they will be 3 times larger than DeepFlavour and by itself be around 1.5% of total miniAOD time, which seems still acceptable.

I don't see any changes in the TagInfos in this PR, but in the profiler
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/Remini_cpu/637
btagbtvdeep::seedingTracksToFeatures is the largest contributor. Please remind me if it's a piece used by DeepFlavour already or if it's a part specific to the needs of DeepVertex/DeepCombined.

leonardogiannini · 2020-11-11T19:05:03Z

I made an additional check with 1000 events, here are the profiles
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/cpu1K
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/mem1K

in particular:
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/cpu1K/17
https://legianni.web.cern.ch/legianni/cgi-bin/igprof-navigator/testMINI+bTag/mem1K/98

the TagInfos are not modified, since the modules are not in standard production. However, seedingTracksToFeatures must be switched on to run the test and compute the taggers. Anyway, DeepVertex and DeepCombined will not be run together other than in BTV workflows, so the extra time should be counted from DeepCombined and seedingTracksToFeatures only.

slava77 · 2020-11-11T19:52:58Z

However, seedingTracksToFeatures must be switched on to run the test and compute the taggers.

I see, indeed in DeepFlavourTagInfoProducer if (run_deepVertex_) { ... btagbtvdeep::seedingTracksToFeatures(.
this apparently triggers a larger cost increase than the inference part .
Some careful inspection and optimization of repeated calls better be done here soon.
This has some nested loop over tracks with some expensive computations repeated.

slava77 · 2020-11-11T20:00:39Z

Some careful inspection and optimization of repeated calls better be done here soon.

I created #32114 to keep track of the progress to improve the TagInfos code

jpata · 2020-11-12T15:18:54Z

+reconstruction

implements DeepVertex and DeepCombined evaluators using ONNX, they are not enabled by default
old TF evaluators are removed
CPU & memory budget have been verified in reMINIAOD and are under control
(needs adding onnx models for DeepVertex and DeeJet+DeepVertex combination cms-data/RecoBTag-Combined#37)

riga · 2020-11-12T18:31:31Z

@jpata

Something that came up in a reco discussion: would we still need a TF evaluator in the future, for e.g. static compilation or GPU evaluation? In this case, it might be premature to remove it, however, the two evaluators should be kept in sync if both are kept in the release. I wonder what's the guidance from the ML prod group @riga @mialiu149?

We are working on the integration of TF GPU into cmsdist, but we cannot give a reliable timeline yet, so no objects from our side to remove the TF evaluator at this point.

santocch · 2020-11-12T19:27:14Z

+1

cmsbuild · 2020-11-12T19:27:41Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2020-11-13T00:12:50Z

+1

leonardogiannini added 4 commits October 29, 2020 16:38

enable DeepCombinedJetTags as supported tagger

0d21c42

move to ONNX and add combination

bf3813a

move producers to ONNX and add DeepCombinedONNXJetTags

7bb2adf

adding cfg to run the DeepVertex and comb. taggers and compare to Dee…

b62be97

…pJet etc.

cmsbuild added this to the CMSSW_11_2_X milestone Oct 29, 2020

cmsbuild added analysis-pending code-checks-pending comparison-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Oct 29, 2020

leonardogiannini mentioned this pull request Oct 29, 2020

adding onnx models for DeepVertex and DeeJet+DeepVertex combination cms-data/RecoBTag-Combined#37

Merged

cmsbuild added code-checks-rejected and removed code-checks-pending labels Oct 29, 2020

ran scram build code-format

c01ee19

cmsbuild added code-checks-pending and removed code-checks-rejected labels Oct 29, 2020

cmsbuild added code-checks-approved and removed code-checks-pending labels Oct 29, 2020

cmsbuild added tests-started and removed tests-pending labels Nov 4, 2020

cmsbuild added tests-approved and removed tests-started labels Nov 11, 2020

cmsbuild added comparison-available and removed comparison-pending labels Nov 11, 2020

slava77 mentioned this pull request Nov 11, 2020

optimize btagbtvdeep::seedingTracksToFeatures re DeepVertex (or deep flavor+vertex combined) tagging #32114

Open

cmsbuild added reconstruction-approved and removed reconstruction-pending labels Nov 12, 2020

cmsbuild removed analysis-pending pending-signatures labels Nov 12, 2020

cmsbuild added analysis-approved fully-signed labels Nov 12, 2020

cmsbuild added orp-approved and removed orp-pending labels Nov 13, 2020

cmsbuild merged commit 90f3ee2 into cms-sw:master Nov 13, 2020

vberta mentioned this pull request Nov 24, 2020

DeepCore (NN jetCore seeding) implementation #32222

Merged

leonardogiannini mentioned this pull request Jan 15, 2021

Deepvertex and DeepJet+DeepVertex combination backport to 10_6_x #32647

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

DeepVertex and DeepJet+DeepVertex combination in release (onnx inference) #31988

DeepVertex and DeepJet+DeepVertex combination in release (onnx inference) #31988

leonardogiannini commented Oct 29, 2020 •

edited

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

santocch commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 •

edited

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 11, 2020

leonardogiannini commented Nov 11, 2020

jpata commented Nov 11, 2020

cmsbuild commented Nov 11, 2020

leonardogiannini commented Nov 11, 2020

slava77 commented Nov 11, 2020 •

edited

leonardogiannini commented Nov 11, 2020 •

edited

slava77 commented Nov 11, 2020

slava77 commented Nov 11, 2020 •

edited

jpata commented Nov 12, 2020 •

edited

riga commented Nov 12, 2020

santocch commented Nov 12, 2020

cmsbuild commented Nov 12, 2020

qliphy commented Nov 13, 2020

DeepVertex and DeepJet+DeepVertex combination in release (onnx inference) #31988

DeepVertex and DeepJet+DeepVertex combination in release (onnx inference) #31988

Conversation

leonardogiannini commented Oct 29, 2020 • edited

PR description:

PR validation:

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

cmsbuild commented Oct 29, 2020

santocch commented Nov 4, 2020

cmsbuild commented Nov 4, 2020 • edited

cmsbuild commented Nov 4, 2020

cmsbuild commented Nov 11, 2020

leonardogiannini commented Nov 11, 2020

jpata commented Nov 11, 2020

cmsbuild commented Nov 11, 2020

leonardogiannini commented Nov 11, 2020

slava77 commented Nov 11, 2020 • edited

leonardogiannini commented Nov 11, 2020 • edited

slava77 commented Nov 11, 2020

slava77 commented Nov 11, 2020 • edited

jpata commented Nov 12, 2020 • edited

riga commented Nov 12, 2020

santocch commented Nov 12, 2020

cmsbuild commented Nov 12, 2020

qliphy commented Nov 13, 2020

leonardogiannini commented Oct 29, 2020 •

edited

cmsbuild commented Nov 4, 2020 •

edited

slava77 commented Nov 11, 2020 •

edited

leonardogiannini commented Nov 11, 2020 •

edited

slava77 commented Nov 11, 2020 •

edited

jpata commented Nov 12, 2020 •

edited