Vulkan ICD file is mounted under /etc/vulkan/icd.d but expected at /usr/share/vulkan/icd.d

[yzq1314]

When running Python packages such as SAPIEN, Vulkan initialization fails with the following warning:
python3 -c 'import sapien; sapien.Scene()'

UserWarning: Failed to find Vulkan ICD file. This is probably due to an incorrect or partial installation of the NVIDIA driver.

After checking, the issue is that SAPIEN (and other Vulkan-based libraries) expect the Vulkan ICD JSON file at:

/usr/share/vulkan/icd.d/nvidia_icd.json

However, in containers using nvidia-container-toolkit, the file is mounted at:

/etc/vulkan/icd.d/nvidia_icd.json

Because of this mismatch, SAPIEN fails to locate the ICD file unless users manually set:

export VK_ICD_FILENAMES=/etc/vulkan/icd.d/nvidia_icd.json

or create a symlink.

Steps to reproduce

Run a container with NVIDIA runtime:

docker run --rm --gpus all -it nvidia/cuda:12.4.1-base-ubuntu22.04 bash

Inside the container:

python3 -c 'import sapien; sapien.Scene()'

Observe warning:

UserWarning: Failed to find Vulkan ICD file. ...

Expected behavior

The Vulkan ICD JSON should be available at the standard location /usr/share/vulkan/icd.d/nvidia_icd.json,
or the container environment should automatically set VK_ICD_FILENAMES accordingly.

Workarounds

Manually export the environment variable:

export VK_ICD_FILENAMES=/etc/vulkan/icd.d/nvidia_icd.json

Or create a symlink:

ln -s /etc/vulkan/icd.d/nvidia_icd.json /usr/share/vulkan/icd.d/nvidia_icd.json

System Info

Container base image: nvidia/cuda:12.4.1-base-ubuntu22.04

Host driver version: 570.195.03

CUDA version: 12.4

Toolkit version: NVIDIA Container Runtime Hook version 1.17.8

Reproducibility: 100%

Proposed solution

Either:

Mount the Vulkan ICD JSON file to both /etc/vulkan/icd.d and /usr/share/vulkan/icd.d,
or

Automatically set VK_ICD_FILENAMES in the container runtime environment.