Add GPU support for ONNXRuntime #36963

hqucms · 2022-02-14T17:57:56Z

PR description:

This is a technical PR that allows the GPU support of ONNXRuntime (cms-sw/cmsdist#6776) to be easily enabled in CMSSW.

Example usage:

  using namespace cms::Ort;
  std::string model_path = edm::FileInPath("PhysicsTools/ONNXRuntime/test/data/model.onnx").fullPath();
  auto session_options = ONNXRuntime::defaultSessionOptions(/*use_cuda=*/true);
  ONNXRuntime rt(model_path, &session_options);

PR validation:

Unit tests are updated to cover all three options.

cmsbuild · 2022-02-14T18:03:41Z

-code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36963/28320

This PR adds an extra 20KB to repository

Code check has found code style and quality issues which could be resolved by applying following patch(s)

code-checks:
https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36963/28320/code-checks.patch
e.g. curl -k https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36963/28320/code-checks.patch | patch -p1
You can also run scram build code-checks to apply code checks directly

cmsbuild · 2022-02-14T18:15:39Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36963/28323

This PR adds an extra 20KB to repository

cmsbuild · 2022-02-14T18:16:00Z

A new Pull Request was created by @hqucms (Huilin Qu) for master.

It involves the following packages:

PhysicsTools/ONNXRuntime (reconstruction)

@jpata, @cmsbuild, @clacaputo, @slava77 can you please review it and eventually sign? Thanks.
@riga this is something you requested to watch as well.
@perrotta, @dpiparo, @qliphy you are the release manager for this.

cms-bot commands are listed here

tvami · 2022-02-14T18:24:51Z

@cmsbuild , please test

cmsbuild · 2022-02-14T23:19:58Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a42efb/22413/summary.html
COMMIT: 3ef28ff
CMSSW: CMSSW_12_3_X_2022-02-14-1100/slc7_amd64_gcc10
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/36963/22413/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 5 differences found in the comparisons
DQMHistoTests: Total files compared: 46
DQMHistoTests: Total histograms compared: 3764435
DQMHistoTests: Total failures: 13
DQMHistoTests: Total nulls: 1
DQMHistoTests: Total successes: 3764399
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.004 KiB( 45 files compared)
DQMHistoSizes: changed ( 312.0 ): 0.004 KiB MessageLogger/Warnings
Checked 193 log files, 42 edm output root files, 46 DQM output files
TriggerResults: no differences found

jpata · 2022-02-15T13:25:04Z

assign heterogeneous

cmsbuild · 2022-02-15T13:25:24Z

New categories assigned: heterogeneous

@fwyzard,@makortel you have been requested to review this Pull request/Issue and eventually sign? Thanks

jpata · 2022-02-15T13:26:38Z

Neat! I'm curious:

how complicated would it be to extend to the Async mode in CMSSW?
now it grabs the whole GPU, how difficult would it be to extend to use streams?

fwyzard · 2022-02-15T13:45:40Z

PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc

+        if (gpu_mode == force_gpu) {
+          throw cms::Exception("RuntimeError") << "No GPU detected, cannot run ONNXRuntime on GPU.";
+        } else {
+          std::cout << "[ONNXRuntime] No GPU detected, will run on CPU." << std::endl;


please don't use std::cout, as it cannot be silenced from the job configuration

Would edm::LogInfo work?

hqucms · 2022-02-15T13:46:54Z

how complicated would it be to extend to the Async mode in CMSSW?

I have never tried to use the async mode in CMSSW -- can you point me to any documentation or example?

now it grabs the whole GPU, how difficult would it be to extend to use streams?

It does not grabs the whole GPU, so in principle it can be used in multiple streams simultaneously I think (to be tested of course).

fwyzard · 2022-02-15T13:47:18Z

PhysicsTools/ONNXRuntime/src/ONNXRuntime.cc

+        if (input_dims[0] != batch_size) {
+          throw cms::Exception("RuntimeError") << "The first element of `input_shapes` (" << input_dims[0]
+                                               << ") does not match the given `batch_size` (" << batch_size << ")";
+        }


this change is unrelated to the use of GPUs, isn't it ?

No it's unrelated to GPU -- just a helpful error message for users.

fwyzard · 2022-02-15T13:53:33Z

What NVIDIA GPU architectures (SM 6.0, 7.0, 7.5, etc.) does ONNX support ?
Is it tied to how we use CUDA in CMSSW, or is it independent from it ?

cmsbuild · 2022-04-22T08:43:55Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-36963/29454

This PR adds an extra 16KB to repository

cmsbuild · 2022-04-22T08:44:17Z

Pull request #36963 was updated. @makortel, @slava77, @clacaputo, @cmsbuild, @fwyzard, @jpata can you please check and sign again.

hqucms · 2022-04-22T08:45:26Z

@makortel The enum class is used implemented in 5eb31fb.
Sorry for the long delay, this somehow slipped through the cracks...

tvami · 2022-04-22T13:48:25Z

@cmsbuild , please test

cmsbuild · 2022-04-22T18:58:34Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a42efb/24133/summary.html
COMMIT: 5eb31fb
CMSSW: CMSSW_12_4_X_2022-04-22-1100/slc7_amd64_gcc10
Additional Tests: GPU
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/36963/24133/install.sh to create a dev area with all the needed externals and cmssw changes.

GPU Comparison Summary

Summary:

No significant changes to the logs found
Reco comparison results: 24 differences found in the comparisons
DQMHistoTests: Total files compared: 4
DQMHistoTests: Total histograms compared: 19874
DQMHistoTests: Total failures: 1195
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 18679
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 3 files compared)
Checked 12 log files, 9 edm output root files, 4 DQM output files
TriggerResults: no differences found

Comparison Summary

@slava77 comparisons for the following workflows were not done due to missing matrix map:

/data/cmsbld/jenkins/workspace/compare-root-files-short-matrix/data/PR-a42efb/39434.75_TTbar_14TeV+2026D88_HLT75e33+TTbar_14TeV_TuneCP5_GenSimHLBeamSpot14+DigiTrigger+RecoGlobal+HLT75e33+HARVESTGlobal

Summary:

No significant changes to the logs found
Reco comparison results: 0 differences found in the comparisons
DQMHistoTests: Total files compared: 49
DQMHistoTests: Total histograms compared: 3695434
DQMHistoTests: Total failures: 2
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 3695410
DQMHistoTests: Total skipped: 22
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 48 files compared)
Checked 205 log files, 45 edm output root files, 49 DQM output files
TriggerResults: no differences found

makortel · 2022-04-22T19:42:34Z

+1

Thanks @hqucms

jpata · 2022-04-25T06:55:24Z

+reconstruction

technical, readiness for CUDA use in ONNXRuntime
no changes

cmsbuild · 2022-04-25T06:55:42Z

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @perrotta, @dpiparo, @qliphy (and backports should be raised in the release meeting by the corresponding L2)

qliphy · 2022-04-25T08:29:44Z

+1

cmsbuild added this to the CMSSW_12_3_X milestone Feb 14, 2022

cmsbuild added code-checks-pending orp-pending pending-signatures reconstruction-pending tests-pending labels Feb 14, 2022

cmsbuild added code-checks-rejected and removed code-checks-pending labels Feb 14, 2022

hqucms force-pushed the dev/ort-gpu branch from c2d3921 to 3ef28ff Compare February 14, 2022 18:10

cmsbuild added code-checks-pending and removed code-checks-rejected labels Feb 14, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Feb 14, 2022

cmsbuild added tests-started and removed tests-pending labels Feb 14, 2022

cmsbuild added tests-approved and removed tests-started labels Feb 14, 2022

cmsbuild added the heterogeneous-pending label Feb 15, 2022

fwyzard reviewed Feb 15, 2022

View reviewed changes

cmsbuild added the tests-pending label Apr 22, 2022

cmsbuild added code-checks-approved and removed code-checks-pending labels Apr 22, 2022

cmsbuild added tests-started and removed tests-pending labels Apr 22, 2022

cmsbuild added tests-approved and removed tests-started labels Apr 22, 2022

cmsbuild added heterogeneous-approved and removed heterogeneous-pending labels Apr 22, 2022

cmsbuild added fully-signed reconstruction-approved and removed reconstruction-pending pending-signatures labels Apr 25, 2022

cmsbuild added orp-approved and removed orp-pending labels Apr 25, 2022

cmsbuild merged commit 863035b into cms-sw:master Apr 25, 2022

cmsbuild mentioned this pull request Apr 25, 2022

Drop TopRex cms-sw/cmsdist#7808

Merged

makortel mentioned this pull request Apr 25, 2022

[Tensorflow] Build with GPU enabled cms-sw/cmsdist#7648

Merged

cmsbuild mentioned this pull request Apr 25, 2022

[CXXMODULE] Update to latest root master changes cms-sw/cmsdist#7819

Merged

davidlange6 mentioned this pull request Sep 15, 2022

Extend onnxruntime gpu interface to producers using onnxruntime #39402

Open

valsdav mentioned this pull request Jan 31, 2023

Add GPU backend option for TensorFlow session #40551

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add GPU support for ONNXRuntime #36963

Add GPU support for ONNXRuntime #36963

hqucms commented Feb 14, 2022 •

edited

cmsbuild commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

tvami commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

jpata commented Feb 15, 2022

cmsbuild commented Feb 15, 2022

jpata commented Feb 15, 2022 •

edited

fwyzard Feb 15, 2022

hqucms Feb 15, 2022

fwyzard Feb 15, 2022 •

edited

hqucms commented Feb 15, 2022

fwyzard Feb 15, 2022

hqucms Feb 15, 2022

fwyzard commented Feb 15, 2022

cmsbuild commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

hqucms commented Apr 22, 2022

tvami commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

makortel commented Apr 22, 2022

jpata commented Apr 25, 2022

cmsbuild commented Apr 25, 2022

qliphy commented Apr 25, 2022

Add GPU support for ONNXRuntime #36963

Add GPU support for ONNXRuntime #36963

Conversation

hqucms commented Feb 14, 2022 • edited

PR description:

PR validation:

cmsbuild commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

tvami commented Feb 14, 2022

cmsbuild commented Feb 14, 2022

Comparison Summary

jpata commented Feb 15, 2022

cmsbuild commented Feb 15, 2022

jpata commented Feb 15, 2022 • edited

fwyzard Feb 15, 2022

Choose a reason for hiding this comment

hqucms Feb 15, 2022

Choose a reason for hiding this comment

fwyzard Feb 15, 2022 • edited

Choose a reason for hiding this comment

hqucms commented Feb 15, 2022

fwyzard Feb 15, 2022

Choose a reason for hiding this comment

hqucms Feb 15, 2022

Choose a reason for hiding this comment

fwyzard commented Feb 15, 2022

cmsbuild commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

hqucms commented Apr 22, 2022

tvami commented Apr 22, 2022

cmsbuild commented Apr 22, 2022

GPU Comparison Summary

Comparison Summary

makortel commented Apr 22, 2022

jpata commented Apr 25, 2022

cmsbuild commented Apr 25, 2022

qliphy commented Apr 25, 2022

hqucms commented Feb 14, 2022 •

edited

jpata commented Feb 15, 2022 •

edited

fwyzard Feb 15, 2022 •

edited