ImportError: A dynamic link library (DLL) initialization routine failed after building source with Visual Studio 17.10

[tianleiwu]

Describe the issue

After building from source, onnxruntime cannot be imported from python in a conda environment.

To reproduce

Upgrade Visual Studio to latest stable version (Version 17.10.2), and build the main branch like the following

build.bat --cmake_generator "Visual Studio 17 2022" --config Release --build_dir build\cuda12 --build_wheel --cmake_extra_defines "CMAKE_CUDA_ARCHITECTURES=89" ^
          --parallel --build_shared_lib --use_cuda --cuda_version "12.5" --cuda_home "C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v12.5" ^
		  --cudnn_home "C:\nvidia\CuDNN\9.1.1.17_cuda12" ^
		  --use_tensorrt --tensorrt_home "C:\nvidia\TensorRT\10.0.1.6.cuda-12.4" --use_tensorrt_builtin_parser ^
                  --parallel 4 --nvcc_threads 1 ^
		  --skip_tests ^
		  --cmake_extra_defines onnxruntime_BUILD_UNIT_TESTS=ON

Then import onnxruntime in python under a conda environment. It shows error:

ImportError: DLL load failed while importing onnxruntime_pybind11_state: A dynamic link library (DLL) initialization routine failed.

In Windows event viewer, I saw an event with information like:

Faulting application name: python.exe, version: 3.10.14150.1013, time stamp: 0x663933b3
Faulting module name: MSVCP140.dll, version: 14.29.30153.0, time stamp: 0x653aa208
Exception code: 0xc0000005
Fault offset: 0x0000000000013080
Faulting process id: 0x0x86A8
Faulting application start time: 0x0x1DABD0F2E51B8F9
Faulting application start time: 0x0x1DABD0F2E51B8F9
Faulting application path: C:\Users\%USERNAME%\.conda\envs\my_env\python.exe
Faulting module path: C:\Users\%USERNAME%\.conda\envs\my_env\MSVCP140.dll

Note that the code has included #21005.

Urgency

No response

Platform

Windows

OS Version

Windows 11 Enterprise 10.0.22631 Build 22631

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

a2b0a69

ONNX Runtime API

Python

Architecture

X64

Execution Provider

Default CPU

Execution Provider Library Version

No response

[yufenglee]

does it persist if you revert the change?

[tianleiwu]

One workaround I found is like the following: copy DLLs from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Redist\MSVC\14.40.33807\onecore\x64\Microsoft.VC143.CRT to my conda environment C:\Users\%USERNAME%\.conda\envs\my_env to overwrite existing DLLs.

Note that installing latest VC Runtime does not work (the downloaded VC Runtime is older than the one installed with latest Visual Studio).

[tianleiwu]

does it persist if you revert the change?

Yes, I encountered the issue before the change.

[yufenglee]

conda uses old vc runtime and it is not compatible with new visual studio compiler. You can run "conda install conda-forge::vs2015_runtime" to upgrade your vc runtime version in conda.

[snnn]

One workaround I found is like the following: copy DLLs from C:\Program Files\Microsoft Visual Studio\2022\Enterprise\VC\Redist\MSVC\14.40.33807\onecore\x64\Microsoft.VC143.CRT to my conda environment C:\Users\%USERNAME%\.conda\envs\my_env to overwrite existing DLLs.

Note that installing latest VC Runtime does not work (the downloaded VC Runtime is older than the one installed with latest Visual Studio).

Or just delete the DLLs in conda's directories.

[snnn]

Faulting module name: MSVCP140.dll, version: 14.29.30153.0, time stamp: 0x653aa208

In theory this should work ... Could it be a different issue?

[snnn]

Oh, @tianleiwu, when you build the code you need to add "--use_binskim_compliant_compile_flags" to your build command, otherwise the changes in #21005 wouldn't take effect. It was because I asked the author to scope the change to a smaller scope that wouldn't effect most people.

[tianleiwu]

Adding --use_binskim_compliant_compile_flags solved the issue.