[10_6_X] ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX

[hqucms]

PR description:

Backport of #28112.

This PR adds the ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX which provides significant speed-up. The previous TensorFlow/MXNet implementations are removed. The new ONNXRuntime-based implementation is enabled by default (as discussed in #30123 (comment)).

Needs:

• [10.6] Backport onnxrt 1.3.0 to 10_6 cmsdist#5902
• Add ONNX models of DeepJet and DeepBoostedJet cms-data/RecoBTag-Combined#25 (already in 10_6_X)

[cmsbuild]

A new Pull Request was created by @hqucms (Huilin Qu) for CMSSW_10_6_X.

It involves the following packages:

PhysicsTools/NanoAOD
PhysicsTools/ONNXRuntime
PhysicsTools/PatAlgos
RecoBTag/Configuration
RecoBTag/ONNXRuntime

@perrotta, @cmsbuild, @fgolf, @slava77, @santocch, @peruzzim can you please review it and eventually sign? Thanks.
@rappoccio, @gouskos, @hatakeyamak, @emilbols, @peruzzim, @seemasharmafnal, @mmarionncern, @ahinzmann, @smoortga, @jdolen, @ferencek, @jdamgov, @nhanvtran, @gkasieczka, @schoef, @andrzejnovak, @clelange, @riga, @JyothsnaKomaragiri, @mverzett, @gpetruc, @mariadalfonso this is something you requested to watch as well.
@silviodonato, @dpiparo you are the release manager for this.

cms-bot commands are listed here

• Backported from ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX #28112

[perrotta]

backport of #28112

[mrodozov]

please test

[cmsbuild]

The tests are being triggered in jenkins.
Tested with other pull request(s) cms-sw/cmsdist#5902
https://cmssdt.cern.ch/jenkins/job/ib-run-pr-tests/6841/console Started: 2020/06/05 15:15

[cmsbuild]

-1

Tested at: 2fffd03

CMSSW: CMSSW_10_6_X_2020-06-04-2300
SCRAM_ARCH: slc7_amd64_gcc700
You can see the results of the tests here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-402a5c/6841/summary.html

I found follow errors while testing this PR

Failed tests: UnitTests RelVals

• Unit Tests:

I found errors in the following unit tests:

---> test runtestPhysicsToolsNanoAOD had ERRORS

• RelVals:

When I ran the RelVals I found an error in the following workflows:
1325.7 step2

runTheMatrix-results/1325.7_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2/step2_TTbar_13_94Xv2NanoAODINPUT+TTbar_13_94Xv2NanoAODINPUT+NANOEDMMC2017_94XMiniAODv2+HARVESTNANOAODMC2017_94XMiniAODv2.log

[cmsbuild]

Comparison not run due to runTheMatrix errors (RelVals and Igprof tests were also skipped)

[cmsbuild]

+1
Tested at: 9671b78
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-291c01/7431/summary.html
CMSSW: CMSSW_10_6_X_2020-06-26-1100
SCRAM_ARCH: slc7_amd64_gcc700

[cmsbuild]

Comparison job queued.

[cmsbuild]

Comparison is ready
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-291c01/7431/summary.html

Comparison Summary:

• No significant changes to the logs found
• Reco comparison results: 3 differences found in the comparisons
• DQMHistoTests: Total files compared: 33
• DQMHistoTests: Total histograms compared: 3212264
• DQMHistoTests: Total failures: 1
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 3211929
• DQMHistoTests: Total skipped: 334
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 32 files compared)
• Checked 137 log files, 14 edm output root files, 33 DQM output files

[santocch]

+1

[slava77]

+1

for #30123 9671b78

• this backports ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX #28112 (the large part) together with more technical updates from Clean BuildFiles in RecoEgamma and PhysicsTools #28779, Suppress warnings from ONNXRuntime #29989, Clean up BuildFiles under PhysicsTools/ #29692, [LLVM10] PhysicsTools fix clang warnings #29581; the TF and MXNet based inference is not moved to ONNXRuntime as in the master version. The implementation of inference is logically the same with some numerical precision differences.
  • based on the discussion leading to [10_6_X] ONNXRuntime-based implementation of DeepJet, DeepAK8 and DeepDoubleX #30123 (comment), the differences at the numerical precision level are acceptable
• jenkins tests pass and comparisons with the baseline show a few single-bin small differences in the b-tag discriminant values.

[peruzzim]

+xpog

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next CMSSW_10_6_X IBs (tests are also fine) and once validation in the development release cycle CMSSW_11_2_X is complete. This pull request will now be reviewed by the release team before it's merged. @silviodonato, @dpiparo (and backports should be raised in the release meeting by the corresponding L2)

[peruzzim]

@hqucms please correct the initial description to make it clear that this is not conditional on the run2_miniAOD_devel modifier anymore

[hqucms]

@hqucms please correct the initial description to make it clear that this is not conditional on the run2_miniAOD_devel modifier anymore

@peruzzim Done!

[silviodonato]

+1