Wrong CUDA version during freezing project with PyTorch

[bw7715]

Describe the bug
I'm trying freeze project with PyTorch 1.11 + CUDA 11.3

But during freezing the project CUDA DLLs are copied from
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\*
instead of
...\venv\lib\site-packages\torch\lib\*

Desktop (please complete the following information):

• Platform information: Windows 10
• cx_Freeze version: 6.13.1
• Python version: 3.9
• PyTorch: 1.11.0+cu113

Part of the cx-freeze output:

creating directory D:\Work\_build\_output_freezer\lib\torch\jit
creating directory D:\Work\_build\_output_freezer\lib\torch\jit\mobile
creating directory D:\Work\_build\_output_freezer\lib\torch\jit\_passes
creating directory D:\Work\_build\_output_freezer\lib\torch\lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\asmjit.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\asmjit.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\asmjit.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\asmjit.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\c10.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\c10.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\c10.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\c10.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\c10_cuda.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\c10_cuda.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\c10_cuda.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\c10_cuda.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\caffe2_nvrtc.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\caffe2_nvrtc.dll
copying C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\nvrtc64_112_0.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\nvrtc64_112_0.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\caffe2_nvrtc.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\caffe2_nvrtc.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\clog.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\clog.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cpuinfo.lib -> D:\Work\_build\_output_freezer\lib\torch\lib\cpuinfo.lib
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cublas64_11.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cublas64_11.dll
copying C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cublasLt64_11.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cublasLt64_11.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cudart64_110.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudart64_110.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cudnn64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn64_8.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cudnn_adv_infer64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn_adv_infer64_8.dll
copying C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudnn_ops_infer64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn_ops_infer64_8.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cudnn_adv_train64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn_adv_train64_8.dll
copying C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.7\bin\cudnn_ops_train64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn_ops_train64_8.dll
copying D:\Work\_build\_venv\lib\site-packages\torch\lib\cudnn_cnn_infer64_8.dll -> D:\Work\_build\_output_freezer\lib\torch\lib\cudnn_cnn_infer64_8.dll

[marcelotduarte]

What are the values of PATH (os.environ["PATH"]) and sys.path?

[marcelotduarte]

Can you provide a minimal sample?
How to get the wheels? Are you using conda?

[marcelotduarte]

I made a patch. To install the latest development build:

pip install --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze --upgrade

[bw7715]

What are the values of PATH (os.environ["PATH"]) and sys.path?

I have both CUDA paths in os.environ and sys.path

C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.7\\bin
C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v11.3\\bin

The v11.7 is before v11.3

[bw7715]

Can you provide a minimal sample? How to get the wheels? Are you using conda?

I'll add sample.
I don't use conda, I install everything be PyPi and PyTorch for wheels with pip install:

--find-links https://download.pytorch.org/whl/cu113/torch_stable.html

[marcelotduarte]

I made a patch. To install the latest development build:

pip install --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze --upgrade

Can you test with this version?

[marcelotduarte]

The v11.7 is before v11.3

You must remove the wrong version from sys.path (and perhaps from PATH).

[bw7715]

The v11.7 is before v11.3

You must remove the wrong version from sys.path (and perhaps from PATH).

I have to use multiple CUDA versions in my projects.
Maybe there is a way to find CUDA DLLs in CUDA_PATH_V**.* variable instead of PATH ?

[marcelotduarte]

I mean, test with the dev version, if that doesn't work, try removing from sys.path and then from PATH.
If you need to remove it, you can use the variables you showed to remove it.
sys.path.remove(CUDA_PATH) and so on, in your setup.py, before the setup() function.

[marcelotduarte]

I made a patch. To install the latest development build:

pip install --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze --upgrade

Can you test with this version?

[bw7715]

This patch did not solve the problem, the files are still copied from the wrong version.

But removing the path with the wrong version of CUDA from PATH helped.

[bw7715]

I made a patch. To install the latest development build:

pip install --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze --upgrade

Can you test with this version?

Yes, I try with version:

6.14.0.dev1

[bw7715]

PS C:\Projects\python_projects\cx_freeze_issues\cuda_dlls_versions\venv\Scripts> .\python.exe -m pip list
Package            Version
------------------ ------------
certifi            2022.12.7
charset-normalizer 2.1.1
cx-Freeze          6.14.0.dev1
cx-Logging         3.1.0
idna               3.4
importlib-metadata 5.2.0
lief               0.12.3
numpy              1.24.0
packaging          22.0
Pillow             9.3.0
pip                22.3.1
requests           2.28.1
setuptools         65.6.3
torch              1.11.0+cu113
torchvision        0.12.0+cu113
typing_extensions  4.4.0
urllib3            1.26.13
wheel              0.38.4
zipp               3.11.0

[bw7715]

Code to reproduce:

requirements.txt

--find-links https://download.pytorch.org/whl/cu113/torch_stable.html

torch==1.11.0+cu113
torchvision==0.12.0+cu113

Python project:

import torch

print(f"PyTorch: version={torch.version.__version__}, cuda={torch.version.cuda}")

Freezer script:

set WORKSPACE=%~dp0

set SCRIPTS_DIRPATH=%WORKSPACE%\venv\Scripts
set PYTHON_EXE_PATH="%SCRIPTS_DIRPATH%\python.exe"
set FREEZER_EXE_PATH="%SCRIPTS_DIRPATH%\cxfreeze.exe"
set TARGET_DIR=out
set TARGET_NAME=project_out.exe
set PACKAGES=torch
set PROJECT_SOURCE_PY_PATH=project.py


%PYTHON_EXE_PATH% -m pip install --pre --extra-index-url https://marcelotduarte.github.io/packages/ cx_Freeze --upgrade

%FREEZER_EXE_PATH% -O -OO --target-dir=%TARGET_DIR% --target-name=%TARGET_NAME% --packages=%PACKAGES% %PROJECT_SOURCE_PY_PATH% > cx_freeze_log.txt

In output file with log cx_freeze_log.txt we can find that dlls from wrong CUDA version are copied if both version v11.3 (correct) and v11.7 (wrong) are installed, are in the system path and the wrong version is before the correct version in sys path..

[bw7715]

If I remove all CUDA paths from system PATH and use develop cx_freeze version then everything is OK.