[1.13] Sets CUDA_MODULE_LOADING to LAZY when not set by the user

[syed-ahmed]

This PR sets CUDA_MODULE_LOADING if it's not set by the user. By default, it sets it to "LAZY".

It was tested using the following commands:

python -c "import torch; tensor=torch.randn(20, 16, 50, 100).cuda(); free, total = torch.cuda.cudart().cudaMemGetInfo(0); print(total-free)"

which shows a memory usage of: 287,047,680 bytes

vs

CUDA_MODULE_LOADING="DEFAULT" python -c "import torch; tensor=torch.randn(20, 16, 50, 100).cuda(); free, total = torch.cuda.cudart().cudaMemGetInfo(0); print(total-free)"

which shows 666,632,192 bytes.

C++ implementation is needed for the libtorch users (otherwise it could have been a pure python functionality).

cc: @ptrblck @ngimel @malfet

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/86509

• 📄 Preview Python docs built from this PR
• 📄 Preview C++ docs built from this PR
• ❓ Need help or want to give feedback on the CI? Visit our office hours

Note: Links to docs will display an error until the docs builds have been completed.

❌ 5 Failures, 3 Pending

As of commit 52f3d32:

The following jobs have failed:

• libtorch-cpu-shared-with-deps-cxx11-abi-test / build
• macos-12-py3-arm64 / test (default, 1, 2, macos-m1-12)
• macos-12-py3-x86-64 / test (default, 2, 2, macos-12)
• win-vs2019-cuda11.6-py3 / test (default, 4, 5, windows.8xlarge.nvidia.gpu)
• win-vs2019-cuda11.6-py3 / test (force_on_cpu, 1, 1, windows.4xlarge)

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[xsacha]

In regards to the C++ implementation, maybe_set_cuda_module_loading could be exposed or run static.
However, you would need to set this variable before any of the CUDA libraries get loaded. This proves tricky when it is libtorch bringing in these dependencies. For this reason, unless the CUDA libraries are all delay loaded, it's probably best that libtorch consumers use an equivalent to this function before loading libtorch.