immich-machine-learning CUDA fail

[Pesc0]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

Immich machine learning container cannot run cuda accelerated ML, see error below. Correctly falls back to CPU and works with that.

Also there's an issue where on boot the container starts before the nvidia driver has loaded, and i have to manually restart the container so that it picks it up. After that the problem occurs

suspect: cuda version mismatch between host and container

The OS that Immich Server is running on

Debian trixie VM in Proxmox

Version of Immich Server

v1.140.1

Version of Immich Mobile App

Platform with the issue

• Server
• Web
• Mobile

Device make and model

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${DATA_LOCATION}:/data
      - ${THUMB_LOCATION}:/data/thumbs
      - ${ENCODED_VIDEO_LOCATION}:/data/encoded-video
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

#    user: 1000:1000
    security_opt:
      # Prevent escalation of privileges after the container is started
      - no-new-privileges:true
    cap_drop:
      # Prevent access to raw network traffic
      - NET_RAW
    logging:
      driver: journald


  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, rocm, openvino, rknn] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, rocm, openvino, openvino-wsl, rknn] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    # Note the lack of an `extends` section
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu

    volumes:
#      - model-cache:/cache
      - /home/immich-user/immich-app/volumes/immich-machine-learning/cache:/cache
      - /home/immich-user/immich-app/volumes/immich-machine-learning/.cache:/.cache
      - /home/immich-user/immich-app/volumes/immich-machine-learning/.config:/.config
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false

#    user: 1000:1000
    security_opt:
      # Prevent escalation of privileges after the container is started
      - no-new-privileges:true
    cap_drop:
      # Prevent access to raw network traffic
      - NET_RAW
    logging:
      driver: journald


  redis:
    container_name: immich_redis
    image: docker.io/valkey/valkey:8-bookworm@sha256:a137a2b60aca1a75130022d6bb96af423fefae4eb55faf395732db3544803280
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

#    command: redis-server --loglevel warning
    volumes:
      - /home/immich-user/immich-app/volumes/redis/data:/data
#    user: 1000:1000
    security_opt:
      # Prevent escalation of privileges after the container is started
      - no-new-privileges:true
    cap_drop:
      # Prevent access to raw network traffic
      - NET_RAW
    logging:
      driver: journald


  database:
    container_name: immich_postgres
    image: ghcr.io/immich-app/postgres:17-vectorchord0.4.3-pgvectors0.3.0@sha256:e5cd9fa99b742c89589c688332028d46b5616586112bfdc03217955980f326eb
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
      # Uncomment the DB_STORAGE_TYPE: 'HDD' var if your database isn't stored on SSDs
      # DB_STORAGE_TYPE: 'HDD'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
#      # SEE https://github.com/docker-library/docs/blob/master/postgres/README.md#arbitrary---user-notes
#      - /etc/passwd:/etc/passwd:ro
    shm_size: 128mb
    restart: always

#    user: 1000:1000
    security_opt:
      # Prevent escalation of privileges after the container is started
      - no-new-privileges:true
    cap_drop:
      # Prevent access to raw network traffic
      - NET_RAW
    logging:
      driver: journald


#volumes:
#  model-cache:

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
DATA_LOCATION=/mnt/storage-immich/data

THUMB_LOCATION=/mnt/unimportant-immich/thumbs
ENCODED_VIDEO_LOCATION=/mnt/unimportant-immich/encoded-video

# The location where your database files are stored. Network shares are not supported for the database
DB_DATA_LOCATION=/mnt/storage-immich/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Rome

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
#DB_PASSWORD=postgres
DB_PASSWORD=****

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. start the containers and attempt to use ML

...

Relevant log output

Sep 13 15:50:18 immich systemd[719]: Started docker-a63982a0e98399f513a3d6a0618540fd2c2633f4aa7dcf63afec83a4cf73c8e1.scope - libcontainer container a63982a0e98399f513a3d6a0618540fd2c2633f4aa7dcf63afec83a4cf73c8e1.
Sep 13 15:50:22 immich a63982a0e983[801]: [09/13/25 13:50:22] INFO     Starting gunicorn 23.0.0                           
Sep 13 15:50:22 immich a63982a0e983[801]: [09/13/25 13:50:22] INFO     Listening at: http://[::]:3003 (7)                 
Sep 13 15:50:22 immich a63982a0e983[801]: [09/13/25 13:50:22] INFO     Using worker: immich_ml.config.CustomUvicornWorker 
Sep 13 15:50:22 immich a63982a0e983[801]: [09/13/25 13:50:22] INFO     Booting worker with pid: 8                         
Sep 13 15:50:32 immich a63982a0e983[801]: [09/13/25 13:50:32] INFO     Started server process [8]                         
Sep 13 15:50:32 immich a63982a0e983[801]: [09/13/25 13:50:32] INFO     Waiting for application startup.                   
Sep 13 15:50:32 immich a63982a0e983[801]: [09/13/25 13:50:32] INFO     Created in-memory cache with unloading after 300s  
Sep 13 15:50:32 immich a63982a0e983[801]:                              of inactivity.                                     
Sep 13 15:50:32 immich a63982a0e983[801]: [09/13/25 13:50:32] INFO     Initialized request thread pool with 2 threads.    
Sep 13 15:50:32 immich a63982a0e983[801]: [09/13/25 13:50:32] INFO     Application startup complete.                      
Sep 13 15:50:47 immich a63982a0e983[801]: [09/13/25 13:50:47] INFO     Loading visual model 'ViT-L-16-SigLIP2-384__webli' 
Sep 13 15:50:47 immich a63982a0e983[801]:                              to memory                                          
Sep 13 15:50:47 immich a63982a0e983[801]: [09/13/25 13:50:47] INFO     Setting execution providers to                     
Sep 13 15:50:47 immich a63982a0e983[801]:                              ['CUDAExecutionProvider', 'CPUExecutionProvider'], 
Sep 13 15:50:47 immich a63982a0e983[801]:                              in descending order of preference                  
Sep 13 15:50:52 immich a63982a0e983[801]: 2025-09-13 13:50:52.395222998 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDA failure 999: unknown error ; GPU=-1 ; hostname=a63982a0e983 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
Sep 13 15:50:52 immich a63982a0e983[801]: *************** EP Error ***************
Sep 13 15:50:52 immich a63982a0e983[801]: EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:59 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:151 Parse Failed to parse provider option "device_id": CUDA failure 999: unknown error ; GPU=-1 ; hostname=a63982a0e983 ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=66 ; expr=cudaGetDeviceCount(&num_devices); 
Sep 13 15:50:52 immich a63982a0e983[801]:  when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
Sep 13 15:50:52 immich a63982a0e983[801]: Falling back to ['CPUExecutionProvider'] and retrying.
Sep 13 15:50:52 immich a63982a0e983[801]: ****************************************
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Shutting down due to inactivity.                   
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Shutting down                                      
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Waiting for application shutdown.                  
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Application shutdown complete.                     
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Finished server process [8]                        
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] ERROR    Worker (pid:8) was sent SIGINT!                    
Sep 13 15:56:25 immich a63982a0e983[801]: [09/13/25 13:56:25] INFO     Booting worker with pid: 98                        
Sep 13 15:56:28 immich a63982a0e983[801]: [09/13/25 13:56:28] INFO     Started server process [98]                        
Sep 13 15:56:28 immich a63982a0e983[801]: [09/13/25 13:56:28] INFO     Waiting for application startup.                   
Sep 13 15:56:28 immich a63982a0e983[801]: [09/13/25 13:56:28] INFO     Created in-memory cache with unloading after 300s  
Sep 13 15:56:28 immich a63982a0e983[801]:                              of inactivity.                                     
Sep 13 15:56:28 immich a63982a0e983[801]: [09/13/25 13:56:28] INFO     Initialized request thread pool with 2 threads.    
Sep 13 15:56:28 immich a63982a0e983[801]: [09/13/25 13:56:28] INFO     Application startup complete.

Additional information

HOST:

uname -a
Linux immich 6.12.43+deb13-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.12.43-1 (2025-08-27) x86_64 GNU/Linux

nvidia-smi
Sat Sep 13 17:14:35 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P4                       On  |   00000000:01:00.0 Off |                  Off |
| N/A   32C    P8              6W /   75W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Wed_Aug_20_01:58:59_PM_PDT_2025
Cuda compilation tools, release 13.0, V13.0.88
Build cuda_13.0.r13.0/compiler.36424714_0

ls /usr/local/cuda/lib64/libcudart.so*
/usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libcudart.so.13  /usr/local/cuda/lib64/libcudart.so.13.0.88

echo $PATH
/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/usr/local/cuda-13.0/bin

echo $LD_LIBRARY_PATH
:/usr/local/cuda-13.0/lib64

docker is running in rootless mode

---

IN CONTAINER (docker exec -it immich_machine_learning bash)

nvidia-smi
Sat Sep 13 15:16:49 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 580.82.07              Driver Version: 580.82.07      CUDA Version: 13.0     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  Tesla P4                       Off |   00000000:01:00.0 Off |                  Off |
| N/A   33C    P8              6W /   75W |       0MiB /   8192MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

nvcc --version
bash: nvcc: command not found

python3 -c "import onnxruntime as ort; print(ort.get_device())"
GPU

ls /usr/local/cuda/lib64/libcudart.so*
/usr/local/cuda/lib64/libcudart.so.12  /usr/local/cuda/lib64/libcudart.so.12.2.140

[Pesc0]

update: downgraded host to 12.2, problem persists

Mon Sep 15 01:44:18 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.261.03             Driver Version: 535.261.03   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla P4                       Off | 00000000:01:00.0 Off |                  Off |
| N/A   37C    P0              22W /  75W |      0MiB /  8192MiB |      1%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2025 NVIDIA Corporation
Built on Tue_May_27_02:21:03_PDT_2025
Cuda compilation tools, release 12.9, V12.9.86
Build cuda_12.9.r12.9/compiler.36037853_0

ls /usr/local/cuda/lib64/libcudart.so*
/usr/local/cuda/lib64/libcudart.so  /usr/local/cuda/lib64/libcudart.so.12  /usr/local/cuda/lib64/libcudart.so.12.9.79

[HappyFrowl]

Have you ever found a solution for this issue? Or gotten to know anything more about it ?

[gauntl3t12]

#21911 (reply in thread)

[PaulePils]

I have a similar problem. My host acts as remote ML with CUDA 13, but the container is only CUDA 12.2.
The ML exits with error code 139.
Is it possible for the container to be updated to CUDA 13?

[gauntl3t12]

The fix is to manually set a variable known as the LD_LIBRARY_PATH. Add this to your docker compose file

    environment:
      LD_LIBRARY_PATH: /usr/local/cuda:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

[amaciuc]

I have a similar problem. My host acts as remote ML with CUDA 13, but the container is only CUDA 12.2.

I have exact the same issue.

    environment:
      LD_LIBRARY_PATH: /usr/local/cuda:/usr/local/nvidia/lib:/usr/local/nvidia/lib64

It helps with nothing. /usr/local/cuda points to the old cuda 12.2, check here:

root@540e47629fce:/usr/src# ls -l /usr/local/cuda
lrwxrwxrwx 1 root root 22 Nov 10  2023 /usr/local/cuda -> /etc/alternatives/cuda
root@540e47629fce:/usr/src# ls -l /etc/alternatives/cuda
lrwxrwxrwx 1 root root 20 Nov 10  2023 /etc/alternatives/cuda -> /usr/local/cuda-12.2

I am using a Quadro P2000 Driver Version: 580.95.05 / CUDA Version: 13.0. Frigate is working with zero issues regarding to GPU, but that is because Frigate doesn't use cudnn. In case of Immich, cudnn is installed natively.

Anyway, as workaround I made a cron job to periodically check the availability of nvidia-smi inside the immich ML container:

*/2 * * * * docker exec immich_machine_learning nvidia-smi > /dev/null 2>&1 || docker restart immich_machine_learning || true

I hope this will help.

Until yesterday I ran immich on 2.2.3, I made upgrade to 2.3.1 along with ML. Maybe this situation has been fixed, will see.

[RWDai]

I have a similar problem. Tesla P4
LD_LIBRARY_PATH is not work

2025-10-24 16:35:06.782219502 [E:onnxruntime:Default, cuda_call.cc:118 CudaCall] CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=e65323cce04d ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=67 ; expr=cudnnConvolutionForward(cudnnHandle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.algo, workspace.get(), Base::s_.workspace_bytes, &beta, Base::s_.y_tensor, Base::s_.y_data); 
2025-10-24 16:35:06.782258233 [E:onnxruntime:, sequential_executor.cc:516 ExecuteKernel] Non-zero status code returned while running FusedConv node. Name:'Conv_0' Status Message: CUDNN failure 5000: CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ; hostname=e65323cce04d ; file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/fused_conv.cc ; line=67 ; expr=cudnnConvolutionForward(cudnnHandle, &alpha, Base::s_.x_tensor, Base::s_.x_data, Base::s_.w_desc, Base::s_.w_data, Base::s_.conv_desc, Base::s_.algo, workspace.get(), Base::s_.workspace_bytes, &beta, Base::s_.y_tensor, Base::s_.y_data); 
[10/24/25 16:35:06] ERROR    Exception in ASGI application                      
                                                                                
                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/immich_ml/main.py:177 in predict       │
                             │                                                 │
                             │   174 │   │   inputs = text                     │
                             │   175 │   else:                                 │
                             │   176 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 177 │   response = await run_inference(inputs │
                             │   178 │   return ORJSONResponse(response)       │
                             │   179                                           │
                             │   180                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:200 in run_inference │
                             │                                                 │
                             │   197 │   │   response[entry["task"]] = output  │
                             │   198 │                                         │
                             │   199 │   without_deps, with_deps = entries     │
                             │ ❱ 200 │   await asyncio.gather(*[_run_inference │
                             │   201 │   if with_deps:                         │
                             │   202 │   │   await asyncio.gather(*[_run_infer │
                             │   203 │   if isinstance(payload, Image):        │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:195 in               │
                             │ _run_inference                                  │
                             │                                                 │
                             │   192 │   │   │   │   message = f"Task {entry[' │
                             │       output of {dep}"                          │
                             │   193 │   │   │   │   raise HTTPException(400,  │
                             │   194 │   │   model = await load(model)         │
                             │ ❱ 195 │   │   output = await run(model.predict, │
                             │   196 │   │   outputs[model.identity] = output  │
                             │   197 │   │   response[entry["task"]] = output  │
                             │   198                                           │
                             │                                                 │
                             │ /usr/src/immich_ml/main.py:213 in run           │
                             │                                                 │
                             │   210 │   if thread_pool is None:               │
                             │   211 │   │   return func(*args, **kwargs)      │
                             │   212 │   partial_func = partial(func, *args, * │
                             │ ❱ 213 │   return await asyncio.get_running_loop │
                             │   214                                           │
                             │   215                                           │
                             │   216 async def load(model: InferenceModel) ->  │
                             │                                                 │
                             │ /usr/local/lib/python3.11/concurrent/futures/th │
                             │ read.py:58 in run                               │
                             │                                                 │
                             │ /usr/src/immich_ml/models/base.py:61 in predict │
                             │                                                 │
                             │    58 │   │   self.load()                       │
                             │    59 │   │   if model_kwargs:                  │
                             │    60 │   │   │   self.configure(**model_kwargs │
                             │ ❱  61 │   │   return self._predict(*inputs, **m │
                             │    62 │                                         │
                             │    63 │   @abstractmethod                       │
                             │    64 │   def _predict(self, *inputs: Any, **mo │
                             │                                                 │
                             │ /usr/src/immich_ml/models/facial_recognition/de │
                             │ tection.py:30 in _predict                       │
                             │                                                 │
                             │   27 │   def _predict(self, inputs: NDArray[np. │
                             │      FaceDetectionOutput:                       │
                             │   28 │   │   inputs = decode_cv2(inputs)        │
                             │   29 │   │                                      │
                             │ ❱ 30 │   │   bboxes, landmarks = self._detect(i │
                             │   31 │   │   return {                           │
                             │   32 │   │   │   "boxes": bboxes[:, :4].round() │
                             │   33 │   │   │   "scores": bboxes[:, 4],        │
                             │                                                 │
                             │ /usr/src/immich_ml/models/facial_recognition/de │
                             │ tection.py:38 in _detect                        │
                             │                                                 │
                             │   35 │   │   }                                  │
                             │   36 │                                          │
                             │   37 │   def _detect(self, inputs: NDArray[np.u │
                             │      NDArray[np.float32]]:                      │
                             │ ❱ 38 │   │   return self.model.detect(inputs)   │
                             │   39 │                                          │
                             │   40 │   def configure(self, **kwargs: Any) ->  │
                             │   41 │   │   self.model.det_thresh = kwargs.pop │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/insightf │
                             │ ace/model_zoo/retinaface.py:224 in detect       │
                             │                                                 │
                             │   221 │   │   det_img = np.zeros( (input_size[1 │
                             │   222 │   │   det_img[:new_height, :new_width,  │
                             │   223 │   │                                     │
                             │ ❱ 224 │   │   scores_list, bboxes_list, kpss_li │
                             │   225 │   │                                     │
                             │   226 │   │   scores = np.vstack(scores_list)   │
                             │   227 │   │   scores_ravel = scores.ravel()     │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/insightf │
                             │ ace/model_zoo/retinaface.py:152 in forward      │
                             │                                                 │
                             │   149 │   │   kpss_list = []                    │
                             │   150 │   │   input_size = tuple(img.shape[0:2] │
                             │   151 │   │   blob = cv2.dnn.blobFromImage(img, │
                             │       (self.input_mean, self.input_mean, self.i │
                             │ ❱ 152 │   │   net_outs = self.session.run(self. │
                             │   153 │   │                                     │
                             │   154 │   │   input_height = blob.shape[2]      │
                             │   155 │   │   input_width = blob.shape[3]       │
                             │                                                 │
                             │ /usr/src/immich_ml/sessions/ort.py:49 in run    │
                             │                                                 │
                             │    46 │   │   input_feed: dict[str, NDArray[np. │
                             │    47 │   │   run_options: Any = None,          │
                             │    48 │   ) -> list[NDArray[np.float32]]:       │
                             │ ❱  49 │   │   outputs: list[NDArray[np.float32] │
                             │       run_options)                              │
                             │    50 │   │   return outputs                    │
                             │    51 │                                         │
                             │    52 │   @property                             │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:22 │
                             │ 0 in run                                        │
                             │                                                 │
                             │    217 │   │   if not output_names:             │
                             │    218 │   │   │   output_names = [output.name  │
                             │    219 │   │   try:                             │
                             │ ❱  220 │   │   │   return self._sess.run(output │
                             │    221 │   │   except C.EPFail as err:          │
                             │    222 │   │   │   if self._enable_fallback:    │
                             │    223 │   │   │   │   print(f"EP Error: {err!s │
                             ╰─────────────────────────────────────────────────╯
                             Fail: [ONNXRuntimeError] : 1 : FAIL : Non-zero     
                             status code returned while running FusedConv node. 
                             Name:'Conv_0' Status Message: CUDNN failure 5000:  
                             CUDNN_STATUS_EXECUTION_FAILED ; GPU=0 ;            
                             hostname=e65323cce04d ;                            
                             file=/onnxruntime_src/onnxruntime/contrib_ops/cuda/
                             fused_conv.cc ; line=67 ;                          
                             expr=cudnnConvolutionForward(cudnnHandle, &alpha,  
                             Base::s_.x_tensor, Base::s_.x_data,                
                             Base::s_.w_desc, Base::s_.w_data,                  
                             Base::s_.conv_desc, Base::s_.algo, workspace.get(),
                             Base::s_.workspace_bytes, &beta, Base::s_.y_tensor,
                             Base::s_.y_data);

[trex2000]

same issue here, running Immich on cuda 13.0 and 580 driver. All other containers work without issues.

[trex2000]

[PiLiPiaPia]

I met the same problem on windows 10, using Docker engine based on WSL2. I found that the directory '/usr/lib/wsl' was not exist in the immich-machine-learning container, which was expected to providing local libcuda.so from host server. Then I bind this path manually by adding following content to docker-compose.yml for immich-machine-learning service:

volumes:
      - /usr/lib/wsl:/usr/lib/wsl:ro

Finally it worked.

[HappyFrowl]

I'm still facing this issue. Interestingly it used to work when I originally set it up 4 months ago.

immich_machine_learning  | 2026-05-14 01:29:16.245315772 [E:onnxruntime:Default, cuda_call.cc:123 CudaCall] CUDA failure 999: unknown error ; GPU=-1 ; hostname=019cf9e2ba4e ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=67 ; expr=cudaGetDeviceCount(&num_devices); 
immich_machine_learning  | *************** EP Error ***************
immich_machine_learning  | EP Error /onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc:60 static onnxruntime::CUDAExecutionProviderInfo onnxruntime::CUDAExecutionProviderInfo::FromProviderOptions(const onnxruntime::ProviderOptions&) [ONNXRuntimeError] : 1 : FAIL : provider_options_utils.h:187 Parse Failed to parse provider option "device_id": CUDA failure 999: unknown error ; GPU=-1 ; hostname=019cf9e2ba4e ; file=/onnxruntime_src/onnxruntime/core/providers/cuda/cuda_execution_provider_info.cc ; line=67 ; expr=cudaGetDeviceCount(&num_devices); 
immich_machine_learning  |  when using ['CUDAExecutionProvider', 'CPUExecutionProvider']
immich_machine_learning  | Falling back to ['CPUExecutionProvider'] and retrying.
immich_machine_learning  | ****************************************

Also same results as amaciuc
on Nov 20, 2025
Cuda version mismatch between host and container

[HappyFrowl]

I managed to fix the cuda version mismatch. I redid the configuration steps in
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html

However, the error remained.