Update gcc 12 builds to CUDA 12.1 #8296

fwyzard · 2023-02-06T22:48:03Z

Update to

CUDA version 12.1.0, with the compatibility NVIDIA drivers version 530.30.02
cuDNN version 8.8.0.121 for CUDA 12.x
ONNX Runtime version 1.14.1

cmsbuild · 2023-02-06T22:48:25Z

A new Pull Request was created by @fwyzard (Andrea Bocci) for branch IB/CMSSW_13_0_X/g12.

@cmsbuild, @smuzaffar, @aandvalenzuela, @iarspider can you please review it and eventually sign? Thanks.
@perrotta, @dpiparo, @rappoccio you are the release manager for this.
cms-bot commands are listed here

fwyzard · 2023-02-06T22:49:51Z

please test for el8_amd64_gcc12

fwyzard · 2023-02-06T22:52:37Z

@smuzaffar how do I test this for a gcc 12 build ?

smuzaffar · 2023-02-07T08:22:13Z

@fwyzard , gcc12 IBs were not built due to failing igprof. We just have merged a fixed for that and triggered an IB to build for gcc12. hopefully in soon we will have gcc12 IB to test this PR

smuzaffar · 2023-02-07T21:18:33Z

please test for el8_amd64_gcc12

cmsbuild · 2023-02-08T01:30:52Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/30472/summary.html
COMMIT: 3c0ab69
CMSSW: CMSSW_13_0_X_2023-02-06-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8296/30472/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:


-- Configuring incomplete, errors occurred!
See also "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.10.0-76ccdb4367437d58968f275617f893f0/build/CMakeFiles/CMakeOutput.log".
See also "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.10.0-76ccdb4367437d58968f275617f893f0/build/CMakeFiles/CMakeError.log".
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.wWvpT6 (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+onnxruntime+1.10.0-76ccdb4367437d58968f275617f893f0
Macro expanded in comment on line 333: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2023-02-08T08:31:54Z

@fwyzard , onnxruntime is still using -Donnxruntime_CUDNN_HOME="${CUDNN_ROOT}"

fwyzard · 2023-02-08T08:37:21Z

please test for el8_amd64_gcc12

cmsbuild · 2023-02-08T08:37:28Z

Pull request #8296 was updated.

cmsbuild · 2023-02-08T10:36:14Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/30489/summary.html
COMMIT: 2804bd9
CMSSW: CMSSW_13_0_X_2023-02-06-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8296/30489/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:


-- Configuring incomplete, errors occurred!
See also "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.10.0-8a886e8eedb1f8d5d94ede1863fe2614/build/CMakeFiles/CMakeOutput.log".
See also "/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.10.0-8a886e8eedb1f8d5d94ede1863fe2614/build/CMakeFiles/CMakeError.log".
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.cuLVXn (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+onnxruntime+1.10.0-8a886e8eedb1f8d5d94ede1863fe2614
Macro expanded in comment on line 333: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

cmsbuild · 2023-04-05T06:28:43Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/31812/summary.html
COMMIT: e3693f4
CMSSW: CMSSW_13_1_X_2023-04-01-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmsdist/8296/31812/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

[707/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/mp11-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/re2/2021-06-01-87d403f0f55e6b0537aad5a4122bd8f5/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm.cc.o -MF CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm.cc.o.d -o CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/embed_layer_norm.cc
[708/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/mp11-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/re2/2021-06-01-87d403f0f55e6b0537aad5a4122bd8f5/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/bias_gelu.cc.o -MF CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/bias_gelu.cc.o.d -o CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/bias_gelu.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/bert/bias_gelu.cc
[709/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/mp11-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/re2/2021-06-01-87d403f0f55e6b0537aad5a4122bd8f5/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/attnlstm/deep_cpu_attn_lstm.cc.o -MF CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/attnlstm/deep_cpu_attn_lstm.cc.o.d -o CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/attnlstm/deep_cpu_attn_lstm.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/contrib_ops/cpu/attnlstm/deep_cpu_attn_lstm.cc
[710/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/mp11-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/re2/2021-06-01-87d403f0f55e6b0537aad5a4122bd8f5/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/core/providers/cpu/tensor/upsample.cc.o -MF CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/core/providers/cpu/tensor/upsample.cc.o.d -o CMakeFiles/onnxruntime_providers.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/core/providers/cpu/tensor/upsample.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-66d9898b747a5a8c1d63fb2b5499c519/onnxruntime-1.14.1/onnxruntime/core/providers/cpu/tensor/upsample.cc
ninja: build stopped: subcommand failed.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.8HlLST (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+onnxruntime+1.14.1-66d9898b747a5a8c1d63fb2b5499c519
Macro expanded in comment on line 336: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2023-04-05T10:27:46Z

please test with cms-sw/cmssw#41118,cms-externals/onnxruntime#9 for el8_amd64_gcc12

cmsbuild · 2023-04-05T14:48:40Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/31833/summary.html
COMMIT: e3693f4
CMSSW: CMSSW_13_1_X_2023-04-04-2300/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8296/31833/install.sh to create a dev area with all the needed externals and cmssw changes.

External Build

I found compilation error when building:

[1122/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-23ccec150fb4f02f6ad3255e137e711a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/mp11-src/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Wno-reorder -Wno-error=sign-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc.o.d -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/tensor/image_scaler.cc
[1123/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/c++ -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-23ccec150fb4f02f6ad3255e137e711a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/mp11-src/include -ffunction-sections -fdata-sections -DCPUINFO_SUPPORTED -O3 -DNDEBUG -fPIC -Wall -Wextra -Wno-deprecated-copy -Wno-nonnull-compare -Wno-reorder -Wno-error=sign-compare -Werror -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/transformers/beam_search.cc.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/transformers/beam_search.cc.o.d -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/transformers/beam_search.cc.o -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/contrib_ops/cuda/transformers/beam_search.cc
[1124/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-23ccec150fb4f02f6ad3255e137e711a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/tensor/resize_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/tensor/resize_impl.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/tensor/resize_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/tensor/resize_impl.cu.o
[1125/1697] /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/bin/nvcc -forward-unknown-to-host-compiler -DCPUINFO_SUPPORTED_PLATFORM=1 -DEIGEN_MPL2_ONLY -DEIGEN_USE_THREADS -DENABLE_CPU_FP16_TRAINING_OPS -DNSYNC_ATOMIC_CPP11 -DONNX_ML=1 -DONNX_NAMESPACE=onnx -DORT_ENABLE_STREAM -DPLATFORM_POSIX -DUSE_CUDA=1 -DUSE_FLASH_ATTENTION=1 -Donnxruntime_providers_cuda_EXPORTS -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/include/onnxruntime/core/session -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/pytorch_cpuinfo-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/google_nsync-src/public -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/abseil_cpp-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/safeint-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/gsl-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/onnx-build -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/protobuf-src/src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/flatbuffers-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cudnn/8.8.0.121-23ccec150fb4f02f6ad3255e137e711a/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/cutlass-src/examples -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/eigen-src -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/cuda/12.1.0-273fdc0c0e67610dc45e0b473a32896d/include -I/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/build/_deps/mp11-src/include -cudart shared --expt-relaxed-constexpr --Werror default-stream-launch -Xcudafe "--diag_suppress=bad_friend_decl" -Xcudafe "--diag_suppress=unsigned_compare_with_zero" -Xcudafe "--diag_suppress=expr_has_no_effect" -O3 -DNDEBUG --generate-code=arch=compute_60,code=[compute_60,sm_60] --generate-code=arch=compute_70,code=[compute_70,sm_70] --generate-code=arch=compute_75,code=[compute_75,sm_75] -Xcompiler=-fPIC --diag-suppress 554 --compiler-options -Wall --compiler-options -Wno-deprecated-copy --compiler-options -Wno-nonnull-compare -Xcompiler -Wno-nonnull-compare --threads "" -Xcompiler -Wno-reorder -Xcompiler -Wno-error=sign-compare -Werror all-warnings -MD -MT CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/math/binary_elementwise_ops_impl.cu.o -MF CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/math/binary_elementwise_ops_impl.cu.o.d -x cu -c /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/math/binary_elementwise_ops_impl.cu -o CMakeFiles/onnxruntime_providers_cuda.dir/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/BUILD/el8_amd64_gcc12/external/onnxruntime/1.14.1-749ef076ed287c891355930e3bbf814d/onnxruntime-1.14.1/onnxruntime/core/providers/cuda/math/binary_elementwise_ops_impl.cu.o
ninja: build stopped: subcommand failed.
error: Bad exit status from /data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/tmp/rpm-tmp.pxWvDk (%build)


RPM build errors:
line 37: It's not recommended to have unversioned Obsoletes: Obsoletes: external+onnxruntime+1.14.1-749ef076ed287c891355930e3bbf814d
Macro expanded in comment on line 336: %{pkginstroot}/${PYTHON3_LIB_SITE_PACKAGES}

smuzaffar · 2023-04-05T14:56:20Z

please test with cms-sw/cmssw#41118,cms-externals/onnxruntime#10 for el8_amd64_gcc12

cmsbuild · 2023-04-05T22:25:37Z

-1

Failed Tests: Build HeaderConsistency
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/31844/summary.html
COMMIT: e3693f4
CMSSW: CMSSW_13_1_X_2023-04-05-1100/el8_amd64_gcc12
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/8296/31844/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/31844/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-7c8bf5/31844/git-merge-result

Build

I found compilation error when building:

>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_13_1_X_2023-04-05-1100/src/DataFormats/Math/test/crossV4_t.cpp
>> Building binary DataFormatscrossV4_t
Copying tmp/el8_amd64_gcc12/src/DataFormats/Math/test/DataFormatscrossV4_t/DataFormatscrossV4_t to productstore area:
>> Compiling  /data/cmsbld/jenkins/workspace/ib-run-pr-tests/CMSSW_13_1_X_2023-04-05-1100/src/DataFormats/Math/test/simpleCholeskyTest.cu
nvcc error   : 'cicc' died due to signal 11 (Invalid memory reference)
gmake: *** [tmp/el8_amd64_gcc12/src/DataFormats/Math/test/simpleCholeskyTest/simpleCholeskyTest.cu.o] Error 1
>> Cuda Device Link tmp/el8_amd64_gcc12/src/DataFormats/Math/test/simpleCholeskyTest/simpleCholeskyTest_cudadlink.o 
nvlink fatal   : Could not open input file 'tmp/el8_amd64_gcc12/src/DataFormats/Math/test/simpleCholeskyTest/simpleCholeskyTest.cu.o' (target: sm_60)
gmake: *** [tmp/el8_amd64_gcc12/src/DataFormats/Math/test/simpleCholeskyTest/simpleCholeskyTest_cudadlink.o] Error 1
>> Building binary simpleCholeskyTest
/data/cmsbld/jenkins/workspace/ib-run-pr-tests/testBuildDir/el8_amd64_gcc12/external/gcc/12.2.0-8db2560e5644f50fe6d5b075da512f77/bin/../lib/gcc/x86_64-redhat-linux-gnu/12.2.1/../../../../x86_64-redhat-linux-gnu/bin/ld.bfd: cannot find tmp/el8_amd64_gcc12/src/DataFormats/Math/test/simpleCholeskyTest/simpleCholeskyTest.cu.o: No such file or directory

cmsbuild · 2023-04-06T09:27:32Z

Pull request #8296 was updated.

smuzaffar · 2023-04-07T15:24:14Z

@fwyzard , I am merging this. Atleast cuda 12 works now with gcc12 but there are still few links errors like [a]

[a]

nvcc error   : 'cicc' died due to signal 11 (Invalid memory reference)

fwyzard changed the base branch from IB/CMSSW_13_0_X/master to IB/CMSSW_13_0_X/g12 February 6, 2023 22:48

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels Feb 6, 2023

cmsbuild added tests-started tests-rejected and removed tests-pending tests-started labels Feb 6, 2023

cmsbuild added tests-started and removed tests-rejected labels Feb 7, 2023

cmsbuild added tests-rejected and removed tests-started labels Feb 8, 2023

fwyzard force-pushed the IB/CMSSW_13_0_X/g12_cuda_12.0.1 branch from 3c0ab69 to 2804bd9 Compare February 8, 2023 08:37

cmsbuild added tests-started and removed tests-rejected labels Feb 8, 2023

cmsbuild added tests-rejected and removed tests-started labels Feb 8, 2023

fwyzard force-pushed the IB/CMSSW_13_0_X/g12_cuda_12.0.1 branch from 2804bd9 to e6c02b0 Compare February 8, 2023 19:46

cmsbuild removed the tests-rejected label Feb 8, 2023

cmsbuild added tests-started and removed tests-rejected labels Apr 4, 2023

cmsbuild added tests-rejected and removed tests-started labels Apr 5, 2023

smuzaffar mentioned this pull request Apr 5, 2023

Update onnxruntime to 1.14.1 #8397

Merged

cmsbuild added tests-started and removed tests-rejected labels Apr 5, 2023

smuzaffar mentioned this pull request Apr 5, 2023

workaround to avoid possible compiler bug cms-externals/onnxruntime#9

Merged

cmsbuild added tests-rejected and removed tests-started labels Apr 5, 2023

cmsbuild added tests-started tests-rejected and removed tests-rejected tests-started labels Apr 5, 2023

smuzaffar mentioned this pull request Apr 6, 2023

[GCC12] Apply patch for spurious Warray-bounds diagnostics #8431

Merged

Update onnxruntime.spec

16a7136

cmsbuild added tests-pending and removed tests-rejected requires-external labels Apr 6, 2023

smuzaffar merged commit 01f65e3 into cms-sw:IB/CMSSW_13_1_X/g12 Apr 7, 2023
2 of 3 checks passed

fwyzard deleted the IB/CMSSW_13_0_X/g12_cuda_12.0.1 branch April 26, 2023 06:36

smuzaffar mentioned this pull request Apr 27, 2023

[DEVEL] CUDA version 12.0.1 and disable Tensorflow GPU #8475

Closed

cmsbuild mentioned this pull request Aug 7, 2023

Use C++20 with more packages #8558

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update gcc 12 builds to CUDA 12.1 #8296

Update gcc 12 builds to CUDA 12.1 #8296

fwyzard commented Feb 6, 2023 •

edited

cmsbuild commented Feb 6, 2023

fwyzard commented Feb 6, 2023

fwyzard commented Feb 6, 2023

smuzaffar commented Feb 7, 2023

smuzaffar commented Feb 7, 2023

cmsbuild commented Feb 8, 2023

smuzaffar commented Feb 8, 2023

fwyzard commented Feb 8, 2023

cmsbuild commented Feb 8, 2023

cmsbuild commented Feb 8, 2023

cmsbuild commented Apr 5, 2023

smuzaffar commented Apr 5, 2023

cmsbuild commented Apr 5, 2023

smuzaffar commented Apr 5, 2023

cmsbuild commented Apr 5, 2023

cmsbuild commented Apr 6, 2023

smuzaffar commented Apr 7, 2023

Update gcc 12 builds to CUDA 12.1 #8296

Update gcc 12 builds to CUDA 12.1 #8296

Conversation

fwyzard commented Feb 6, 2023 • edited

cmsbuild commented Feb 6, 2023

fwyzard commented Feb 6, 2023

fwyzard commented Feb 6, 2023

smuzaffar commented Feb 7, 2023

smuzaffar commented Feb 7, 2023

cmsbuild commented Feb 8, 2023

External Build

smuzaffar commented Feb 8, 2023

fwyzard commented Feb 8, 2023

cmsbuild commented Feb 8, 2023

cmsbuild commented Feb 8, 2023

External Build

cmsbuild commented Apr 5, 2023

External Build

smuzaffar commented Apr 5, 2023

cmsbuild commented Apr 5, 2023

External Build

smuzaffar commented Apr 5, 2023

cmsbuild commented Apr 5, 2023

Build

cmsbuild commented Apr 6, 2023

smuzaffar commented Apr 7, 2023

fwyzard commented Feb 6, 2023 •

edited