Having issues building the environment.

[subtera745]

I've been trying to get this to work on my local. I'm running Ubuntu 22.04. Nvidia-smi 560.35.03 Cuda Version: 12.6.

I keep getting this error, no matter what I do. Which is confusing, because I was able to get this working initially, and then something required the appropriate system user credentials, so I thought to reinstall the environment to fix the user credentials i had set wrong the first time. Upon reinstallation of this environment, i was not prompted to provide a different uid or gid, as I believe I was prompted to provide this the first time, I could be wrong. regardless, I ended up removing and re-installing my nvidia drivers, and cuda drivers, making sure to install the proper toolkits for them as well. I believe at least. Half the time I feel a bit gaslit, because I still may have messed things up. I have removed and cleaned away the docker images, containers and volumes. I have added specific verbiage to my docker daemon files as listed here NVIDIA/nvidia-docker#838 . I have updated the project Dockerfile to reflect my environment nvidia and cuda versioning. However, i understand that runtime cuda drivers are indicative of being managed by python.

All that to say, should I be trying to enter the docker environment to sort out the appropriate versioning of the python runtime packages? Should I try and modify the user_script.bash to install updated runtime python packages? This reddit post mentions that they had to reinstall the appropriate runtime packages https://www.reddit.com/r/comfyui/comments/1cw33fy/comfy_installation_broken_for_anyone_else/

Should I utilize that to match the appropriate versioning I have installed? I do not know how to proceed. I have been trying to get this working for three days. Please advise.

comfyui-nvidia | -- COMFYUI_PATH: /comfy/mnt/ComfyUI
comfyui-nvidia | == You will need to run ComfyUI-Manager a first time for the configuration file to be generated, we can not attempt to update its
security level yet
comfyui-nvidia | == Container directory: /comfy/mnt/ComfyUI
comfyui-nvidia | == Checking for user script: /comfy/mnt/user_script.bash
comfyui-nvidia | ===================
comfyui-nvidia | == Running ComfyUI
comfyui-nvidia | -- Running: python3 ./main.py --listen 0.0.0.0 --disable-auto-launch
comfyui-nvidia | [START] Security scan
comfyui-nvidia | [DONE] Security scan
comfyui-nvidia | ## ComfyUI-Manager: installing dependencies done.
comfyui-nvidia | ** ComfyUI startup time: 2024-09-29 00:24:19.563793
comfyui-nvidia | ** Platform: Linux
comfyui-nvidia | ** Python version: 3.10.12 (main, Jul 29 2024, 16:56:48) [GCC 11.4.0]
comfyui-nvidia | ** Python executable: /comfy/mnt/venv/bin/python3
comfyui-nvidia | ** ComfyUI Path: /comfy/mnt/ComfyUI
comfyui-nvidia | ** Log path: /comfy/mnt/ComfyUI/comfyui.log
comfyui-nvidia |
comfyui-nvidia | Prestartup times for custom nodes:
comfyui-nvidia | 0.2 seconds: /comfy/mnt/ComfyUI/custom_nodes/ComfyUI-Manager
comfyui-nvidia |
comfyui-nvidia | Traceback (most recent call last):
comfyui-nvidia | File "/comfy/mnt/ComfyUI/./main.py", line 90, in
comfyui-nvidia | import execution
comfyui-nvidia | File "/comfy/mnt/ComfyUI/execution.py", line 13, in
comfyui-nvidia | import nodes
comfyui-nvidia | File "/comfy/mnt/ComfyUI/nodes.py", line 21, in
comfyui-nvidia | import comfy.diffusers_load
comfyui-nvidia | File "/comfy/mnt/ComfyUI/comfy/diffusers_load.py", line 3, in
comfyui-nvidia | import comfy.sd
comfyui-nvidia | File "/comfy/mnt/ComfyUI/comfy/sd.py", line 5, in
comfyui-nvidia | from comfy import model_management
comfyui-nvidia | File "/comfy/mnt/ComfyUI/comfy/model_management.py", line 143, in
comfyui-nvidia | total_vram = get_total_memory(get_torch_device()) / (1024 * 1024)
comfyui-nvidia | File "/comfy/mnt/ComfyUI/comfy/model_management.py", line 112, in get_torch_device
comfyui-nvidia | return torch.device(torch.cuda.current_device())
comfyui-nvidia | File "/comfy/mnt/venv/lib/python3.10/site-packages/torch/cuda/init.py", line 878, in current_device
comfyui-nvidia | _lazy_init()
comfyui-nvidia | File "/comfy/mnt/venv/lib/python3.10/site-packages/torch/cuda/init.py", line 314, in _lazy_init
comfyui-nvidia | torch._C._cuda_init()
comfyui-nvidia | RuntimeError: No CUDA GPUs are available
comfyui-nvidia | ComfyUI failed or exited with an error
comfyui-nvidia | == user -- uid: 1024 / gid: 1024
comfyui-nvidia | Wrong GID (1024), exiting
comfyui-nvidia exited with code 0

edit: I also have no idea how to change the GID and UID that's been provided. I changed the Dockerfile and init.bash to uid / gid 1000 respectively, but still. it keeps being set as 1024.

[mmartial]

To confirm: are you running this within Docker?
I see RuntimeError: No CUDA GPUs are available which makes me wonder if it is an issue with the NVIDIA container toolkit?
Can you check this post https://blg.gkr.one/20240404-u24_nvidia_docker_podman/ and confirm that

docker run --rm --runtime=nvidia --gpus all nvidia/cuda:11.6.2-base-ubuntu20.04 nvidia-smi

returns a valid output of nvidia-smi
This will tell you if the runtime is functional. You can modify the default-runtime after.

For the UID/GID, see the WANTED_UID and WANTED_GID environment variables detailed in the "Running the container" docker run section of the GitHub's README.md

PS: is 560 finally official?

[subtera745]

Thank you for your apt response.

It seems that I need to restart completely from scratch to match the appropriate versioning.

nvidia-smi by itself returns

Sat Sep 28 20:36:58 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 560.35.03 Driver Version: 560.35.03 CUDA Version: 12.6 |
|-----------------------------------------+------------------------+----------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+========================+======================|
| 0 NVIDIA GeForce RTX 4080 On | 00000000:01:00.0 On | N/A |
| 0% 36C P8 13W / 320W | 828MiB / 16376MiB | 7% Default |
| | | N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=========================================================================================|
| 0 N/A N/A 6167 G /usr/lib/xorg/Xorg 545MiB |
| 0 N/A N/A 6350 G /usr/bin/gnome-shell 48MiB |
| 0 N/A N/A 8445 G ...seed-version=20240927-160657.442000 205MiB |
| 0 N/A N/A 9260 G gnome-control-center 3MiB |
+-----------------------------------------------------------------------------------------+

however within the docker container by utilizing the command provided returns
Failed to initialize NVML: Unknown Error

560 I believe is official for Ubuntu 24.04. However. I do not know if it is for Ubuntu 22.04, and within the nvidia documentation I cannot find right now, I remember that the Ubuntu 22.04 versioning did not go that high yet in the document, so I do not fully know if it is official. I just know that it was the provided LTS available as an install without providing a specific version number.

It seems that the current path forward is to wipe nvidia and cuda, their respective toolkits, and install fresh to ensure the correct versioning is applied.

Thank you for your time, and I will let you know if this line of action resolves the issue.

[mmartial]

Can you reboot the host?
I have seen the NVML error sometimes happens when the kernel and the container runtime are not in sync with the installed version of the driver

[subtera745]

So again, thank you for your apt response. Terribly sorry for my late one as lots of life things happening right now.
I uninstalled my nvidia and cuda drivers and toolkits and reinstalled them. I downgraded to nvidia driver 550 and cuda 12.4 . While following the podman tutorial linked in the readme, I came across an error with this command.

podman run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi
Error: stat nvidia.com/gpu=all: no such file or directory

Upon researching, I found that a common issue was that after so long, within 48 hours, probably less, docker stops working with the gpus. This stack overflow post however seems to have helped resolve the issue.

https://stackoverflow.com/questions/72932940/failed-to-initialize-nvml-unknown-error-in-docker-after-few-hours

However, I don't think I would have stumbled upon this had I not received your input to make sure the basics were in fact, all lined up. Especially since it was only at the end of the podman tutorial that I found this out.

So I believe I can say with confidence, that the issue I had is now resolved. Thank you so much for your time.