Exception in ASGI application - [GPU] out of GPU resources

[effsee00]

The bug

When running Face Detection on a system configured with OpenVino ML - the immich_machine_learning container will eventually throw an exception complaining out of GPU resources. immich_machine_learning does not recover.

This has been recreated triggering "ALL" Face Detection, but the exception can appear at an indeterminate time after some success with face detection.

The OS that Immich Server is running on

DSM 7.2.1-69057 Update 5 | Docker version 20.10.23

Version of Immich Server

v1.117.0

Version of Immich Mobile App

v1.117.0 build.162

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the stack.env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - ${EXTERNAL_LOCATION}:/data/external:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-openvino
   # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
   #   file: hwaccel.ml.yml
   #   service: openvino # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    device_cgroup_rules:
      - 'c 189:* rmw'
    devices:
      - /dev/dri:/dev/dri
    volumes:
      - model-cache:/cache
      - /dev/bus/usb:/dev/bus/usb
      - ${EXTERNAL_LOCATION}:/data/external:ro
    env_file:
      - stack.env
    restart: always
    healthcheck:
      disable: false

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:2d1463258f2764328496376f5d965f20c6a67f66ea2b06dc42af351f75248792
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the stack.env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: pg_isready --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' || exit 1; Chksum="$$(psql --dbname='${DB_DATABASE_NAME}' --username='${DB_USERNAME}' --tuples-only --no-align --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')"; echo "checksum failure count is $$Chksum"; [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
#      start_interval: 30s
#      start_period: 5m
    command: ["postgres", "-c", "shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]
    restart: always

volumes:
  model-cache:

Your .env content

UPLOAD_LOCATION=/volume1/data/immich/library
EXTERNAL_LOCATION=/volume1/homes
DB_DATA_LOCATION=/volume2/docker/immich/postgres
TZ=Europe/London
IMMICH_VERSION=release
DB_PASSWORD=postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich
IMMICH_LOG_LEVEL=debug
IMMICH_AUTO_CREATE_ALBUM=true
MACHINE_LEARNING_WORKER_TIMEOUT=600

Reproduction steps

1. Using an external library of ~8000 images / videos,
2. Start Face Detection job "ALL"
3. Wait for exception
   ...

Relevant log output

[10/07/24 16:07:31] DEBUG    Checking for inactivity...
[10/07/24 16:07:41] DEBUG    Checking for inactivity...
[10/07/24 16:07:42] DEBUG    Setting model format to onnx
[10/07/24 16:07:42] INFO     Loading detection model 'antelopev2' to memory
[10/07/24 16:07:42] DEBUG    Available ORT providers:
                             {'OpenVINOExecutionProvider',
                             'CPUExecutionProvider'}
[10/07/24 16:07:42] DEBUG    Available OpenVINO devices: ['CPU', 'GPU']
[10/07/24 16:07:42] INFO     Setting execution providers to
                             ['OpenVINOExecutionProvider',
[10/07/24 16:07:42] DEBUG    Setting execution provider options to
                             [{'device_type': 'GPU', 'precision': 'FP32',
                             'cache_dir':
                             '/cache/facial-recognition/antelopev2/detection/ope
                             nvino'}, {'arena_extend_strategy':
[10/07/24 16:07:42] DEBUG    Setting execution_mode to ORT_SEQUENTIAL
[10/07/24 16:07:42] DEBUG    Setting inter_op_num_threads to 0
[10/07/24 16:07:42] DEBUG    Setting intra_op_num_threads to 0
[10/07/24 16:07:43] DEBUG    Setting model format to onnx
[10/07/24 16:07:43] INFO     Loading recognition model 'antelopev2' to memory
[10/07/24 16:07:43] DEBUG    Available ORT providers:
                             {'OpenVINOExecutionProvider',
                             'CPUExecutionProvider'}
[10/07/24 16:07:43] DEBUG    Available OpenVINO devices: ['CPU', 'GPU']
[10/07/24 16:07:43] INFO     Setting execution providers to
                             ['OpenVINOExecutionProvider',
[10/07/24 16:07:43] DEBUG    Setting execution provider options to
                             [{'device_type': 'GPU', 'precision': 'FP32',
[10/07/24 16:07:43] DEBUG    Setting execution_mode to ORT_SEQUENTIAL
[10/07/24 16:07:43] DEBUG    Setting inter_op_num_threads to 0
[10/07/24 16:07:43] DEBUG    Setting intra_op_num_threads to 0
2024-10-07 16:07:46.107496327 [E:onnxruntime:, inference_session.cc:2105 operator()] Exception during initialization: /onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:87 onnxruntime::openvino_ep::OVExeNetwork onnxruntime::openvino_ep::OVCore::CompileModel(std::shared_ptr<const ov::Model>&, std::string&, ov::AnyMap&, const std::string&) [OpenVINO-EP]  Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_0Exception from src/inference/src/cpp/core.cpp:107:
Exception from src/inference/src/dev/plugin.cpp:53:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cpp:201:
[GPU] out of GPU resources




[10/07/24 16:07:46] ERROR    Exception in ASGI application

                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/app/main.py:152 in predict             │
                             │                                                 │
                             │   149 │   │   inputs = text                     │
                             │   150 │   else:                                 │
                             │   151 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 152 │   response = await run_inference(inputs │
                             │   153 │   return ORJSONResponse(response)       │
                             │   154                                           │
                             │   155                                           │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ entries = (                                 │ │
                             │ │           │   [                             │ │
                             │ │           │   │   {                         │ │
                             │ │           │   │   │   'name': 'antelopev2', │ │
                             │ │           │   │   │   'task':               │ │
                             │ │           'facial-recognition',             │ │
                             │ │           │   │   │   'type': 'detection',  │ │
                             │ │           │   │   │   'options': {          │ │
                             │ │           │   │   │   │   'minScore': 0.7   │ │
                             │ │           │   │   │   }                     │ │
                             │ │           │   │   }                         │ │
                             │ │           │   ],                            │ │
                             │ │           │   [                             │ │
                             │ │           │   │   {                         │ │
                             │ │           │   │   │   'name': 'antelopev2', │ │
                             │ │           │   │   │   'task':               │ │
                             │ │           'facial-recognition',             │ │
                             │ │           │   │   │   'type':               │ │
                             │ │           'recognition',                    │ │
                             │ │           │   │   │   'options': {}         │ │
                             │ │           │   │   }                         │ │
                             │ │           │   ]                             │ │
                             │ │           )                                 │ │
                             │ │   image = b'\xff\xd8\xff\xe2\x01\xf0ICC_PR… │ │
                             │ │           \x00\x00mntrRGB XYZ               │ │
                             │ │           \x07\xe2\x00\x03\x00\x14\x00\t\x… │ │
                             │ │  inputs = <PIL.JpegImagePlugin.JpegImageFi… │ │
                             │ │           image mode=RGB size=1915x1440 at  │ │
                             │ │           0x7F24F4665950>                   │ │
                             │ │    text = None                              │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/main.py:177 in run_inference       │
                             │ │                  │   │   │   'type':        │ │
                             │ │                  'detection',               │ │
                             │ │                  │   │   │   'options': {   │ │
                             │ │                  │   │   │   │              │ │
                             │ │                  'minScore': 0.7            │ │
                             │ │                  │   │   │   }              │ │
                             │ │                  │   │   }                  │ │
                             │ │                  │   }                      │ │
                             │ │                  ]                          │ │
                             │ │   without_deps = [                          │ │
                             │ │                  │   {                      │ │
                             │   197 │   │   │   raise HTTPException(500, f"Fa │
                             │   198 │   │   with lock:                        │
                             │   199 │   │   │   try:                          │
                             │   201 │   │   │   except FileNotFoundError as e │
                             │   202 │   │   │   │   if model.model_format ==  │
                             │   203 │   │   │   │   │   raise e               │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ model = <app.models.facial_recognition.rec… │ │
                             │ │         object at 0x7f24f4a36e50>           │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/base.py:53 in load          │
                             │ │    self = <app.models.facial_recognition.r… │ │
                             │ │           object at 0x7f24f4a36e50>         │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/facial_recognition/recognit │
                             │ ion.py:28 in _load                              │
                             │                                                 │
                             │   25 │   │   self.batch = self.model_format ==  │
                             │   26 │                                          │
                             │   27 │   def _load(self) -> ModelSession:       │
                             │ ❱ 28 │   │   session = self._make_session(self. │
                             │   29 │   │   if self.batch and str(session.get_ │
                             │   30 │   │   │   self._add_batch_axis(self.mode │
                             │   31 │   │   │   session = self._make_session(s │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ self = <app.models.facial_recognition.reco… │ │
                             │ │        object at 0x7f24f4a36e50>            │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/base.py:110 in              │
                             │ _make_session                                   │
                             │                                                 │
                             │   107 │   │   │   case ".armnn":                │
                             │   108 │   │   │   │   session: ModelSession = A │
                             │   109 │   │   │   case ".onnx":                 │
                             │ ❱ 110 │   │   │   │   session = OrtSession(mode │
                             │   111 │   │   │   case _:                       │
                             │   112 │   │   │   │   raise ValueError(f"Unsupp │
                             │   113 │   │   return session                    │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ model_path = PosixPath('/cache/facial-reco… │ │
                             │ │       self = <app.models.facial_recognitio… │ │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │       model_path = PosixPath('/cache/facia… │ │
                             │ │ provider_options = None                     │ │
                             │ │        providers = None                     │ │
                             │ │             self = <app.sessions.ort.OrtSe… │ │
                             │ │                    object at                │ │
                             │ │                    0x7f24f4a34c50>          │ │
                             │ │     sess_options = None                     │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:41 │
                             │ 9 in __init__                                   │
                             │                                                 │
                             │    416 │   │   disabled_optimizers = kwargs.get │
                             │    417 │   │                                    │
                             │    418 │   │   try:                             │
                             │ ❱  419 │   │   │   self._create_inference_sessi │
                             │        disabled_optimizers)                     │
                             │    420 │   │   except (ValueError, RuntimeError │
                             │    421 │   │   │   if self._enable_fallback:    │
                             │    422 │   │   │   │   try:                     │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ disabled_optimizers = None                  │ │
                             │ │              kwargs = {}                    │ │
                             │ │       path_or_bytes = '/cache/facial-recog… │ │
                             │ │    provider_options = [                     │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
                             │ │                       'device_type': 'GPU', │ │
                             │ │                       │   │   'precision':  │ │
                             │ │                       'FP32',               │ │
                             │ │                       │   │   'cache_dir':  │ │
                             │ │                       '/cache/facial-recog… │ │
                             │ │                       │   },                │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
                             │ │                       'arena_extend_strate… │ │
                             │ │                       'kSameAsRequested'    │ │
                             │ │                       │   }                 │ │
                             │ │                       ]                     │ │
                             │ │           providers = [                     │ │
                             │ │                       │                     │ │
                             │ │                       'OpenVINOExecutionPr… │ │
                             │ │                       │                     │ │
                             │ │                       'CPUExecutionProvide… │ │
                             │ │                       ]                     │ │
                             │ │                self = <onnxruntime.capi.on… │ │
                             │ │                       object at             │ │
                             │ │                       0x7f24f4a34d50>       │ │
                             │ │        sess_options = <onnxruntime.capi.on… │ │
                             │ │                       object at             │ │
                             │ │                       0x7f24f4664970>       │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:49 │
                             │ 1 in _create_inference_session                  │
                             │                                                 │
                             │    488 │   │   │   disabled_optimizers = set(di │
                             │    489 │   │                                    │
                             │    490 │   │   # initialize the C++ InferenceSe │
                             │ ❱  491 │   │   sess.initialize_session(provider │
                             │    492 │   │                                    │
                             │    493 │   │   self._sess = sess                │
                             │    494 │   │   self._sess_options = self._sess. │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ available_providers = [                     │ │
                             │ │                       │                     │ │
                             │ │                       'OpenVINOExecutionPr… │ │
                             │ │                       │                     │ │
                             │ │                       'CPUExecutionProvide… │ │
                             │ │                       ]                     │ │
                             │ │ disabled_optimizers = set()                 │ │
                             │ │    provider_options = [                     │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
                             │ │                       'device_type': 'GPU', │ │
                             │ │                       │   │   'precision':  │ │
                             │ │                       'FP32',               │ │
                             │ │                       │   │   'cache_dir':  │ │
                             │ │                       '/cache/facial-recog… │ │
                             │ │                       │   },                │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
                             │ │                       'arena_extend_strate… │ │
                             │ │                       'kSameAsRequested'    │ │
                             │ │                       │   }                 │ │
                             │ │                       ]                     │ │
                             │ │           providers = [                     │ │
                             │ │                       │                     │ │
                             │ │                       'OpenVINOExecutionPr… │ │
                             │ │                       │                     │ │
                             │ │                       'CPUExecutionProvide… │ │
                             │ │                       ]                     │ │
                             │ │                self = <onnxruntime.capi.on… │ │
                             │ │                       object at             │ │
                             │ │                       0x7f24f4a34d50>       │ │
                             │ │                sess = <onnxruntime.capi.on… │ │
                             │ │                       object at             │ │
                             │ │                       0x7f24f46659f0>       │ │
                             │ │     session_options = <onnxruntime.capi.on… │ │
                             │ │                       object at             │ │
                             │ │                       0x7f24f4664970>       │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             ╰─────────────────────────────────────────────────╯
                             RuntimeException: [ONNXRuntimeError] : 6 :
                             RUNTIME_EXCEPTION : Exception during
                             initialization:
                             /onnxruntime/onnxruntime/core/providers/openvino/ov
                             _interface.cc:87
                             onnxruntime::openvino_ep::OVExeNetwork
                             onnxruntime::openvino_ep::OVCore::CompileModel(std:
                             :shared_ptr<const ov::Model>&, std::string&,
                             ov::AnyMap&, const std::string&) [OpenVINO-EP]
                             Exception while Loading Network for graph:
                             OpenVINOExecutionProvider_OpenVINO-EP-subgraph_4_0E
                             xception from src/inference/src/cpp/core.cpp:107:
                             Exception from src/inference/src/dev/plugin.cpp:53:
                             Exception from
                             src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cp
                             p:201:
                             [GPU] out of GPU resources




[10/07/24 16:07:47] INFO     Attempt #2 to load recognition model 'antelopev2'
                             to memory
[10/07/24 16:07:47] DEBUG    Available ORT providers:
                             {'OpenVINOExecutionProvider',
                             'CPUExecutionProvider'}
[10/07/24 16:07:47] DEBUG    Available OpenVINO devices: ['CPU', 'GPU']
[10/07/24 16:07:47] INFO     Setting execution providers to
                             ['OpenVINOExecutionProvider',
                             'CPUExecutionProvider'], in descending order of
[10/07/24 16:07:47] DEBUG    Setting execution provider options to
                             [{'device_type': 'GPU', 'precision': 'FP32',
[10/07/24 16:07:47] DEBUG    Setting execution_mode to ORT_SEQUENTIAL
[10/07/24 16:07:47] DEBUG    Setting inter_op_num_threads to 0
[10/07/24 16:07:47] DEBUG    Setting intra_op_num_threads to 0
2024-10-07 16:07:49.566109514 [E:onnxruntime:, inference_session.cc:2105 operator()] Exception during initialization: /onnxruntime/onnxruntime/core/providers/openvino/ov_interface.cc:87 onnxruntime::openvino_ep::OVExeNetwork onnxruntime::openvino_ep::OVCore::CompileModel(std::shared_ptr<const ov::Model>&, std::string&, ov::AnyMap&, const std::string&) [OpenVINO-EP]  Exception while Loading Network for graph: OpenVINOExecutionProvider_OpenVINO-EP-subgraph_5_0Exception from src/inference/src/cpp/core.cpp:107:
Exception from src/inference/src/dev/plugin.cpp:53:
Exception from src/plugins/intel_gpu/src/runtime/ocl/ocl_engine.cpp:201:
[GPU] out of GPU resources

[10/07/24 16:07:49] ERROR    Exception in ASGI application

                             ╭─────── Traceback (most recent call last) ───────╮
                             │ /usr/src/app/main.py:152 in predict             │
                             │                                                 │
                             │   149 │   │   inputs = text                     │
                             │   150 │   else:                                 │
                             │   151 │   │   raise HTTPException(400, "Either  │
                             │ ❱ 152 │   response = await run_inference(inputs │
                             │   153 │   return ORJSONResponse(response)       │
                             │   154                                           │
                             │   155                                           │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ entries = (                                 │ │
                             │ │           │   [                             │ │
                             │ │           │   │   {                         │ │
                             │ │           │   │   │   'name': 'antelopev2', │ │
                             │ │           │   │   │   'task':               │ │
                             │ │           'facial-recognition',             │ │
                             │ │           │   │   │   'type': 'detection',  │ │
                             │ │           │   │   │   'options': {          │ │
                             │ │           │   │   │   │   'minScore': 0.7   │ │
                             │ │           │   │   │   }                     │ │
                             │ │           │   │   }                         │ │
                             │ │           │   ],                            │ │
                             │ │           │   [                             │ │
                             │ │           │   │   {                         │ │
                             │ │           │   │   │   'name': 'antelopev2', │ │
                             │ │           │   │   │   'task':               │ │
                             │ │           'facial-recognition',             │ │
                             │ │           │   │   │   'type':               │ │
                             │ │           'recognition',                    │ │
                             │ │           │   │   │   'options': {}         │ │
                             │ │           │   │   }                         │ │
                             │ │           │   ]                             │ │
                             │ │           )                                 │ │
                             │ │   image = b'\xff\xd8\xff\xe2\x01\xf0ICC_PR… │ │
                             │ │           \x00\x00mntrRGB XYZ               │ │
                             │ │           \x07\xe2\x00\x03\x00\x14\x00\t\x… │ │
                             │ │  inputs = <PIL.JpegImagePlugin.JpegImageFi… │ │
                             │ │           image mode=RGB size=1915x1440 at  │ │
                             │ │           0x7F24F4B99950>                   │ │
                             │ │    text = None                              │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/main.py:177 in run_inference       │
                             │                                                 │
                             │   174 │   without_deps, with_deps = entries     │
                             │   178 │   if isinstance(payload, Image):        │
                             │   179 │   │   response["imageHeight"], response │
                             │   180                                           │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ _run_inference = <function                  │ │
                             │ │                  run_inference.<locals>._r… │ │
                             │ │                  at 0x7f24f4c009a0>         │ │
                             │ │        entries = (                          │ │
                             │ │                  │   [                      │ │
                             │ │                  │   │   {                  │ │
                             │ │                  │   │   │   'name':        │ │
                             │ │                  'antelopev2',              │ │
                             │ │                  │   │   │   'task':        │ │
                             │ │         func = <function                    │ │
                             │ │                load.<locals>._load at       │ │
                             │ │                0x7f24f4c03880>              │ │
                             │ │       kwargs = {}                           │ │
                             │ │ partial_func = functools.partial(<function  │ │
                             │ │                load.<locals>._load at       │ │
                             │ │                0x7f24f4c03880>,             │ │
                             │ │                <app.models.facial_recognit… │ │
                             │ │                object at 0x7f24f4a36e50>)   │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │   201 │   │   │   except FileNotFoundError as e │
                             │   202 │   │   │   │   if model.model_format ==  │
                             │   203 │   │   │   │   │   raise e               │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │         object at 0x7f24f4a36e50>           │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/base.py:53 in load          │
                             │                                                 │
                             │    50 │   │   self.download()                   │
                             │    51 │   │   attempt = f"Attempt #{self.load_a │
                             │       else "Loading"                            │
                             │    52 │   │   log.info(f"{attempt} {self.model_ │
                             │       '{self.model_name}' to memory")           │
                             │ ❱  53 │   │   self.session = self._load()       │
                             │    54 │   │   self.loaded = True                │
                             │    55 │                                         │
                             │    56 │   def predict(self, *inputs: Any, **mod │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ attempt = 'Attempt #2 to load'              │ │
                             │ │    self = <app.models.facial_recognition.r… │ │
                             │ │           object at 0x7f24f4a36e50>         │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/facial_recognition/recognit │
                             │ ion.py:28 in _load                              │
                             │                                                 │
                             │   25 │   │   self.batch = self.model_format ==  │
                             │   26 │                                          │
                             │   27 │   def _load(self) -> ModelSession:       │
                             │ ❱ 28 │   │   session = self._make_session(self. │
                             │   29 │   │   if self.batch and str(session.get_ │
                             │   30 │   │   │   self._add_batch_axis(self.mode │
                             │   31 │   │   │   session = self._make_session(s │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ self = <app.models.facial_recognition.reco… │ │
                             │ │        object at 0x7f24f4a36e50>            │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/models/base.py:110 in              │
                             │ _make_session                                   │
                             │                                                 │
                             │   107 │   │   │   case ".armnn":                │
                             │   108 │   │   │   │   session: ModelSession = A │
                             │   109 │   │   │   case ".onnx":                 │
                             │ ❱ 110 │   │   │   │   session = OrtSession(mode │
                             │   111 │   │   │   case _:                       │
                             │   112 │   │   │   │   raise ValueError(f"Unsupp │
                             │   113 │   │   return session                    │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ model_path = PosixPath('/cache/facial-reco… │ │
                             │ │       self = <app.models.facial_recognitio… │ │
                             │ │              object at 0x7f24f4a36e50>      │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /usr/src/app/sessions/ort.py:28 in __init__     │
                             │                                                 │
                             │    25 │   │   self.providers = providers if pro │
                             │    26 │   │   self.provider_options = provider_ │
                             │       self._provider_options_default            │
                             │    27 │   │   self.sess_options = sess_options  │
                             │       self._sess_options_default                │
                             │ ❱  28 │   │   self.session = ort.InferenceSessi │
                             │    29 │   │   │   self.model_path.as_posix(),   │
                             │    30 │   │   │   providers=self.providers,     │
                             │    31 │   │   │   provider_options=self.provide │
                             │                                                 │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │       model_path = PosixPath('/cache/facia… │ │
                             │ │ provider_options = None                     │ │
                             │ │        providers = None                     │ │
                             │ │             self = <app.sessions.ort.OrtSe… │ │
                             │ │                    object at                │ │
                             │ │                    0x7f24f4b9b010>          │ │
                             │ │     sess_options = None                     │ │
                             │ ╰─────────────────────────────────────────────╯ │
                             │                                                 │
                             │ /opt/venv/lib/python3.11/site-packages/onnxrunt │
                             │ ime/capi/onnxruntime_inference_collection.py:41 │
                             │ 9 in __init__                                   │
                             │                                                 │
                             │    416 │   │   disabled_optimizers = kwargs.get │
                             │    417 │   │                                    │
                             │    418 │   │   try:                             │
                             │ ❱  419 │   │   │   self._create_inference_sessi │
                             │        disabled_optimizers)                     │
                             │    420 │   │   except (ValueError, RuntimeError │
                             │    421 │   │   │   if self._enable_fallback:    │
                             │    422 │   │   │   │   try:                     │
                             │ ╭────────────────── locals ───────────────────╮ │
                             │ │ disabled_optimizers = None                  │ │
                             │ │              kwargs = {}                    │ │
                             │ │       path_or_bytes = '/cache/facial-recog… │ │
                             │ │    provider_options = [                     │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
                             │ │                       'device_type': 'GPU', │ │
                             │ │                       │   │   'precision':  │ │
                             │ │                       'FP32',               │ │
                             │ │                       │   │   'cache_dir':  │ │
                             │ │                       '/cache/facial-recog… │ │
                             │ │                       │   {                 │ │
                             │ │                       │   │                 │ │
[10/07/24 16:07:51] DEBUG    Checking for inactivity...
[10/07/24 16:08:01] DEBUG    Checking for inactivity...

Additional information

Host hardware:
Synology DS423+
CPU: INTEL Celeron J4125
Physical memory: 18 GB

[AdoKevin]

The same issue occurred while face detection with my device CPU INTEL N5095. I changed the model from buffalo_l to buffalo_m then this exception stopped. I believe thi is caused by the poor performance of my integrated GPU.

One more thing to metion is when this exception was thown, the undo queue got stuck, but the ml container still reported as healthy, perhaps some exeption handling logic should be investigated.

[lscorcia]

Same observation here. After an inference crash all work by the ML container stopped, but the container always says healthy. If it were possible to discover this condition, we could restart the container if unhealthy.

[mertalev]

This is not a bug as far as I can tell, but just a quirk of OpenVINO’s memory allocation. OpenVINO seems to handle batching in a way that’s efficient for speed but not memory usage. As it encounters batches of different sizes, it makes allocations specific to each particular batch size to quickly reuse when it encounters that batch size again. In this case, a batch is the number of faces in an image. This means that as it encounters images with varying numbers of faces (especially ones with many faces), it can use more memory over time. This increase plateaus when the model stops seeing new batch sizes.

I can add an env to explicitly disable batching for cases when the GPU doesn’t have enough memory to handle this.

[effsee00]

Thank you for the explanation.

In the interim would there be something I could add into the container health check to monitor for this state, so a restart is actioned?

[mertalev]

If you can access the machine learning logs within the health check, you can make it search for this error. I don't think the container will be automatically restarted if it's marked unhealthy, though, so you'd have to apply one of these methods to get it to actually restart.

[lscorcia]

I'm not sure batching is involved in this issue. I do have the same problem and I'm on a Synology NAS (Intel J3355, Apollo Lake). If I read correctly the docs for batching, it is enabled automatically only if the GPU reports OPTIMAL_BATCH_SIZE and MAX_BATCH_SIZE > 1 (https://docs.openvino.ai/2024/openvino-workflow/running-inference/inference-devices-and-modes/gpu-device.html#automatic-batching). In my case, the GPU reports:

[ INFO ]                Immutable: OPTIMAL_BATCH_SIZE : 1
[ INFO ]                Immutable: MAX_BATCH_SIZE : 1

I had this RAM exhaustion issue for a while and I also upgraded my device from 10 to 16GB of RAM but all it did was delay the issue a bit. It shows up quickly during Face detection.

Honestly I don't think it's an Immich problem - I feel like something weird is going on between the ONNX runtime and the model, a leak or maybe fragmentation issues. As someone who knows next to nothing about ML, the only workaround I can imagine adding to Immich is a parameter to unload the runtime after N inference requests. In my case, N ~= 400.

[mertalev]

That's for the automatic batching feature that internally combines multiple singular inputs into a batch. We don't use that - we explicitly submit a batch of faces to the model.

[mertalev]

I believe you can make it periodically auto-restart a worker by setting GUNICORN_CMD_ARGS=--max-requests 400 (or however many)