Describe the bug
Environment:
Host OS: Bluefin (Version: latest-42.20250518 / FROM Fedora Silverblue 42)
Distrobox Version: distrobox: 1.8.1.2
Container Engine: Podman 5.5.0 (as reported by Distrobox init log: engine=podman-5.5.0)
NVIDIA Driver Version (Host): 570.144 (CUDA Version 12.8 reported by nvidia-smi)
NVIDIA Container Toolkit Version (Host): 1.17.7 (libnvidia-container1-1.17.7, nvidia-container-toolkit-1.17.7, etc.)
Test Container Image: ubuntu:22.04 (Issue also observed with a Fedora 41 based container previously)
Problem Description:
When creating a Distrobox container using the --nvidia flag on the described host system, the NVIDIA container environment is not fully or correctly established inside the container. This occurs despite Distrobox's startup sequence reporting "[ OK ]" for "Setting up host's nvidia integration...".
Specifically, two key components of a standard NVIDIA-enabled container environment are missing:
Missing ld.so.conf.d configuration for NVIDIA: The directory /etc/ld.so.conf.d/ inside the Distrobox container does not contain an nvidia.conf file (or a similarly named file like 00-nvcr-.conf) that would typically be created by the NVIDIA Container Toolkit to point the dynamic linker to the mounted host driver libraries.
Missing NVIDIA Environment Variables: Standard NVIDIA_ environment variables (e.g., NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES, NVIDIA_REQUIRE_CUDA) are not set or inherited inside the Distrobox container. Running env | grep NVIDIA returns no output.
Symptoms and Impact:
While basic tools like nvidia-smi function correctly within the Distrobox container (likely due to Distrobox performing direct bind-mounts of NVIDIA binaries and some libraries into standard paths like /lib/x86_64-linux-gnu/), applications requiring a fully configured NVIDIA runtime environment may fail.
In this specific case, PyTorch was installed with CUDA 12.4 support (torch-2.6.0+cu124). Basic PyTorch CUDA operations succeed. However, PyTorch's Triton JIT compiler (version 3.2.0) fails when attempting to compile and load its internal cuda_utils.so module, resulting in an error: ImportError: /path/to/.triton/cache/.../cuda_utils.so: undefined symbol: cuModuleGetFunction
This error persists even after ensuring:
The host libcuda.so.1 (found at /lib/x86_64-linux-gnu/libcuda.so.1 inside the container via ldconfig -p) contains the cuModuleGetFunction symbol (confirmed with nm -D).
The Triton cache is cleared before each run.
The issue is triggered when a custom node in ComfyUI (comfyui-kjnodes using sageattention) invokes a Triton kernel.
A similar pattern of incomplete NVIDIA integration (empty libcuda.so symlinks) was observed in a previous, separate troubleshooting session for DaVinci Resolve on a similar host/Distrobox setup, further suggesting a systemic issue with Distrobox's NVIDIA setup on this host configuration.
Steps to Reproduce:
On a host system matching the environment description (Bluefin/Fedora Atomic, Podman 5.5.0, NVIDIA driver 570.144, NVIDIA Container Toolkit 1.17.7).
Create a new Distrobox container:
Bash
distrobox create --nvidia --name test-nvidia-issue --image ubuntu:22.04
Enter the container:
Bash
distrobox enter test-nvidia-issue
Check for NVIDIA environment setup:
ls -l /etc/ld.so.conf.d/
Observation: No nvidia.conf or 00-nvcr-*.conf file is present.
env | grep NVIDIA
Observation: No NVIDIA environment variables are listed.
Comparison with Direct Podman Run (Key Diagnostic Finding):
When a similar container is run directly with Podman on the same host, the NVIDIA environment is set up more completely by the NVIDIA Container Toolkit:
Bash
Run on the host system
podman run --rm --security-opt label=disable --device nvidia.com/gpu=all ubuntu:22.04 sh -c "echo '--- ld.so.conf.d ---'; ls -l /etc/ld.so.conf.d/; echo '--- NVIDIA Env Vars ---'; env | grep NVIDIA"
Result of direct Podman run:
An NVIDIA-specific .conf file (e.g., 00-nvcr-.conf) is present in /etc/ld.so.conf.d/.
At least one NVIDIA_ environment variable is set (e.g., NVIDIA_VISIBLE_DEVICES=void, though other variables like NVIDIA_DRIVER_CAPABILITIES might still be expected for a full setup).
This discrepancy indicates that the issue likely lies in how Distrobox implements the --nvidia functionality or interacts with Podman and the NVIDIA OCI runtime hooks, rather than a fundamental problem with the host's Podman or NVIDIA Container Toolkit installation itself (as those seem to work correctly for direct Podman runs).
Host Journal Logs during Distrobox init:
Host journal logs for the distrobox-init process (from journalctl) show Distrobox performing many mount --rbind operations for individual NVIDIA libraries and binaries from /run/host/... to standard paths within the container. However, these logs do not show any errors during the "Setting up host's nvidia integration..." phase, nor do they show the creation of the /etc/ld.so.conf.d/nvidia.conf file.
Expected Behavior:
A Distrobox container created with the --nvidia flag should have a fully configured NVIDIA runtime environment, including:
The automatic creation of an appropriate nvidia.conf (or similar) file in /etc/ld.so.conf.d/ by the NVIDIA container runtime hooks, ensuring the dynamic linker is correctly configured.
The injection of standard NVIDIA_* environment variables (e.g., NVIDIA_VISIBLE_DEVICES=all, NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics,display,video).
Possible Root Cause:
The Distrobox NVIDIA integration scripts may be incompletely replicating or interfering with the setup steps normally performed by the NVIDIA OCI runtime hooks when used directly with Podman, particularly concerning linker configuration and environment variable injection. This incomplete environment then causes issues for applications like Triton that rely on a standard NVIDIA container setup for JIT compilation and dynamic linking of CUDA driver APIs.
What did you expect to happen?
I expected 2 + 2 to equal 4, but instead 2 + 2 equaled 6!
Output of bootc status
Output of groups
Extra information or context
No response
Describe the bug
Environment:
Host OS: Bluefin (Version: latest-42.20250518 / FROM Fedora Silverblue 42)
Distrobox Version: distrobox: 1.8.1.2
Container Engine: Podman 5.5.0 (as reported by Distrobox init log: engine=podman-5.5.0)
NVIDIA Driver Version (Host): 570.144 (CUDA Version 12.8 reported by nvidia-smi)
NVIDIA Container Toolkit Version (Host): 1.17.7 (libnvidia-container1-1.17.7, nvidia-container-toolkit-1.17.7, etc.)
Test Container Image: ubuntu:22.04 (Issue also observed with a Fedora 41 based container previously)
Problem Description:
When creating a Distrobox container using the --nvidia flag on the described host system, the NVIDIA container environment is not fully or correctly established inside the container. This occurs despite Distrobox's startup sequence reporting "[ OK ]" for "Setting up host's nvidia integration...".
Specifically, two key components of a standard NVIDIA-enabled container environment are missing:
Missing ld.so.conf.d configuration for NVIDIA: The directory /etc/ld.so.conf.d/ inside the Distrobox container does not contain an nvidia.conf file (or a similarly named file like 00-nvcr-.conf) that would typically be created by the NVIDIA Container Toolkit to point the dynamic linker to the mounted host driver libraries.
Missing NVIDIA Environment Variables: Standard NVIDIA_ environment variables (e.g., NVIDIA_VISIBLE_DEVICES, NVIDIA_DRIVER_CAPABILITIES, NVIDIA_REQUIRE_CUDA) are not set or inherited inside the Distrobox container. Running env | grep NVIDIA returns no output.
Symptoms and Impact:
While basic tools like nvidia-smi function correctly within the Distrobox container (likely due to Distrobox performing direct bind-mounts of NVIDIA binaries and some libraries into standard paths like /lib/x86_64-linux-gnu/), applications requiring a fully configured NVIDIA runtime environment may fail.
In this specific case, PyTorch was installed with CUDA 12.4 support (torch-2.6.0+cu124). Basic PyTorch CUDA operations succeed. However, PyTorch's Triton JIT compiler (version 3.2.0) fails when attempting to compile and load its internal cuda_utils.so module, resulting in an error: ImportError: /path/to/.triton/cache/.../cuda_utils.so: undefined symbol: cuModuleGetFunction
This error persists even after ensuring:
The host libcuda.so.1 (found at /lib/x86_64-linux-gnu/libcuda.so.1 inside the container via ldconfig -p) contains the cuModuleGetFunction symbol (confirmed with nm -D).
The Triton cache is cleared before each run.
The issue is triggered when a custom node in ComfyUI (comfyui-kjnodes using sageattention) invokes a Triton kernel.
A similar pattern of incomplete NVIDIA integration (empty libcuda.so symlinks) was observed in a previous, separate troubleshooting session for DaVinci Resolve on a similar host/Distrobox setup, further suggesting a systemic issue with Distrobox's NVIDIA setup on this host configuration.
Steps to Reproduce:
On a host system matching the environment description (Bluefin/Fedora Atomic, Podman 5.5.0, NVIDIA driver 570.144, NVIDIA Container Toolkit 1.17.7).
Create a new Distrobox container:
Bash
distrobox create --nvidia --name test-nvidia-issue --image ubuntu:22.04
Enter the container:
Bash
distrobox enter test-nvidia-issue
Check for NVIDIA environment setup:
ls -l /etc/ld.so.conf.d/
Observation: No nvidia.conf or 00-nvcr-*.conf file is present.
env | grep NVIDIA
Observation: No NVIDIA environment variables are listed.
Comparison with Direct Podman Run (Key Diagnostic Finding):
When a similar container is run directly with Podman on the same host, the NVIDIA environment is set up more completely by the NVIDIA Container Toolkit:
Bash
Run on the host system
podman run --rm --security-opt label=disable --device nvidia.com/gpu=all ubuntu:22.04 sh -c "echo '--- ld.so.conf.d ---'; ls -l /etc/ld.so.conf.d/; echo '--- NVIDIA Env Vars ---'; env | grep NVIDIA"
Result of direct Podman run:
An NVIDIA-specific .conf file (e.g., 00-nvcr-.conf) is present in /etc/ld.so.conf.d/.
At least one NVIDIA_ environment variable is set (e.g., NVIDIA_VISIBLE_DEVICES=void, though other variables like NVIDIA_DRIVER_CAPABILITIES might still be expected for a full setup).
This discrepancy indicates that the issue likely lies in how Distrobox implements the --nvidia functionality or interacts with Podman and the NVIDIA OCI runtime hooks, rather than a fundamental problem with the host's Podman or NVIDIA Container Toolkit installation itself (as those seem to work correctly for direct Podman runs).
Host Journal Logs during Distrobox init:
Host journal logs for the distrobox-init process (from journalctl) show Distrobox performing many mount --rbind operations for individual NVIDIA libraries and binaries from /run/host/... to standard paths within the container. However, these logs do not show any errors during the "Setting up host's nvidia integration..." phase, nor do they show the creation of the /etc/ld.so.conf.d/nvidia.conf file.
Expected Behavior:
A Distrobox container created with the --nvidia flag should have a fully configured NVIDIA runtime environment, including:
The automatic creation of an appropriate nvidia.conf (or similar) file in /etc/ld.so.conf.d/ by the NVIDIA container runtime hooks, ensuring the dynamic linker is correctly configured.
The injection of standard NVIDIA_* environment variables (e.g., NVIDIA_VISIBLE_DEVICES=all, NVIDIA_DRIVER_CAPABILITIES=compute,utility,graphics,display,video).
Possible Root Cause:
The Distrobox NVIDIA integration scripts may be incompletely replicating or interfering with the setup steps normally performed by the NVIDIA OCI runtime hooks when used directly with Podman, particularly concerning linker configuration and environment variable injection. This incomplete environment then causes issues for applications like Triton that rely on a standard NVIDIA container setup for JIT compilation and dynamic linking of CUDA driver APIs.
What did you expect to happen?
I expected 2 + 2 to equal 4, but instead 2 + 2 equaled 6!
Output of
bootc statusOutput of
groupsExtra information or context
No response