Smart Search doesn't work. Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

[wlodzimierz-dzis]

The bug

Hello team,

After updating to latest verstion v1.125.7 smart serach stopped working. I'm using Immich server on Synology NAS as a host and remote ML with microservices on PC for remote jobs. Both remote and host have the same versions.

Failed to search smart (Immich Server Error)

Server version

Immich server logs

[Nest] 32 - 02/01/2025, 6:09:58 PM WARN [Api:MachineLearningRepositoryxx39mh3a] Machine learning request to "http://192.168.50.217:3003" failed: fetch failed
[Nest] 32 - 02/01/2025, 6:10:09 PM WARN [Api:MachineLearningRepositoryxx39mh3a] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed
[Nest] 32 - 02/01/2025, 6:10:09 PM ERROR [Api:ErrorInterceptor~xx39mh3a] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs
at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)
at process.processTicksAndRejections (node:internal/process/task_queues:105:5)
at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:61:26)
at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:60:27)

Immich remote ML logs

[02/01/25 15:19:38] INFO Starting gunicorn 23.0.0
[02/01/25 15:19:38] INFO Listening at: http://[::]:3003 (9)
[02/01/25 15:19:38] INFO Using worker: app.config.CustomUvicornWorker
[02/01/25 15:19:38] INFO Booting worker with pid: 10
[02/01/25 15:19:43] INFO Started server process [10]
[02/01/25 15:19:43] INFO Waiting for application startup.
[02/01/25 15:19:43] INFO Created in-memory cache with unloading after 300s
of inactivity.
[02/01/25 15:19:43] INFO Initialized request thread pool with 12 threads.
[02/01/25 15:19:43] INFO Application startup complete.
[02/01/25 15:46:37] INFO Starting gunicorn 23.0.0
[02/01/25 15:46:37] INFO Listening at: http://[::]:3003 (9)
[02/01/25 15:46:37] INFO Using worker: app.config.CustomUvicornWorker
[02/01/25 15:46:37] INFO Booting worker with pid: 10
[02/01/25 15:46:43] INFO Started server process [10]
[02/01/25 15:46:43] INFO Waiting for application startup.
[02/01/25 15:46:43] INFO Created in-memory cache with unloading after 300s
of inactivity.
[02/01/25 15:46:43] INFO Initialized request thread pool with 12 threads.
[02/01/25 15:46:43] INFO Application startup complete.
[02/01/25 15:50:16] INFO Loading visual model 'ViT-B-32__openai' to memory
[02/01/25 15:50:16] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],
in descending order of preference
[02/01/25 15:50:19] INFO Loading detection model 'buffalo_l' to memory
[02/01/25 15:50:19] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],
in descending order of preference
[02/01/25 15:50:22] INFO Loading recognition model 'buffalo_l' to memory
[02/01/25 15:50:22] INFO Setting execution providers to
['CUDAExecutionProvider', 'CPUExecutionProvider'],

The OS that Immich Server is running on

Synology Nas as a host, Windows 11 with Nvidia GPU as a remote ML

Version of Immich Server

v.1.125.7

Version of Immich Mobile App

v.1.125.7

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

=================================================HOST================================

services:
  immich-redis:
    image: redis
    container_name: Immich-REDIS
    hostname: immich-redis
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD-SHELL", "redis-cli ping || exit 1"]
    user: 1026:100
    environment:
      - TZ=Europe/Warsaw
    ports:
      - 6379:6379
    volumes:
      - /volume1/docker/immich/redis:/data:rw
    restart: on-failure:5

  immich-db:
    image: tensorchord/pgvecto-rs:pg16-v0.2.0
    container_name: Immich-DB
    hostname: immich-db
    security_opt:
      - no-new-privileges:true
    healthcheck:
      test: ["CMD", "pg_isready", "-q", "-d", "immich", "-U", "immichuser"]
      interval: 10s
      timeout: 5s
      retries: 5
    volumes:
      - /volume1/docker/immich/db:/var/lib/postgresql/data:rw
    ports:
      - 5433:5432
    environment:
      - TZ=Europe/Warsaw
      - POSTGRES_DB=immich
      - POSTGRES_USER=immichuser
      - POSTGRES_PASSWORD=immichpw
    restart: on-failure:5

  immich-server:
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    container_name: Immich-SERVER
    hostname: immich-server
    user: 1026:100
    security_opt:
      - no-new-privileges:true
    env_file:
      - stack.env
    ports:
      - 2283:2283
    volumes:
      - /volume1/docker/immich/upload:/usr/src/app/upload:rw
    restart: on-failure:5
    depends_on:
      immich-redis:
        condition: service_healthy
      immich-db:
        condition: service_started

  immich-machine-learning:
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    container_name: Immich-LEARNING
    hostname: immich-machine-learning
    user: 1026:100
    security_opt:
      - no-new-privileges:true
    env_file:
      - stack.env
    volumes:
      - /volume1/docker/immich/upload:/usr/src/app/upload:rw
      - /volume1/docker/immich/cache:/cache:rw
      - /volume1/docker/immich/matplotlib:/matplotlib:rw
    environment:
      - MPLCONFIGDIR=/matplotlib
    restart: on-failure:5
    depends_on:
      immich-db:
        condition: service_started
        

=================================================REMOTE==========================================


services:
  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
                - compute
                - video
    volumes:
      - immich-uploads:/usr/src/app/upload:rw
      - immich-external-library:/usr/src/app/external/me:rw
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - stack.env
    environment:
      - PUID=1026
      - PGID=100
    restart: always
  
  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}-cuda
    ports:
      - 3003:3003
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities:
                - gpu
    env_file:
      - stack.env
    environment:
      - PUID=1026
      - PGID=100
    volumes:
      - 'C:\\Users\\pc\\model-cache:/cache:rw'
    restart: always

volumes:
  immich-uploads:
    driver_opts:
        type: "nfs"
        o: "addr=IP,nfsvers=3"
        device: ":volume1/docker/immich/upload"
  immich-external-library:
    driver_opts:
        type: "nfs"
        o: "addr=IP,nfsvers=3"
        device: ":volume1/data/..."

Your .env content

NODE_ENV=production
TZ=Europe/Warsaw
DB_HOSTNAME=immich-db
DB_USERNAME=immichuser
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich-redis
LOG_LEVEL=log
DISABLE_REVERSE-GEOCODING=false
REVERSE_GEOCODING_PRECISION=3
PUBLIC_LOGIN_PAGE_MESSAGE=-
IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001
IMMICH_MACHINE_LEARNING_URL=http://immich-machine-learning:3003
IMMICH_WORKERS_INCLUDE=api




============================================HOST=====================================================

DB_HOSTNAME=IP
REDIS_HOSTNAME=IP
DB_USERNAME=immichuser
DB_DATABASE_NAME=immich
DB_PORT=5433
TZ=Europe/Warsaw
IMMICH_VERSION=release
REDIS_PORT=6379
IMMICH_WORKERS_EXCLUDE=api

Reproduction steps

1.Search in the smart search
2.You will get an error
3.
...

Relevant log output

[Nest] 32  - 02/01/2025, 5:10:17 PM    WARN [Api:MachineLearningRepository~bhi5w349] Machine learning request to "http://192.168.50.217:3003" failed: fetch failed

[Nest] 32  - 02/01/2025, 5:10:28 PM    WARN [Api:MachineLearningRepository~bhi5w349] Machine learning request to "http://immich-machine-learning:3003" failed: fetch failed

[Nest] 32  - 02/01/2025, 5:10:28 PM   ERROR [Api:ErrorInterceptor~bhi5w349] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-B-32__openai"}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:38:15)

    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:61:26)

    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:60:27)

Additional information

One important thing to mention, after rerunning Smart Search and Face Detection jobs I can see remote GPU in use in task manager and PC is reachable from NAS.

[SlavenIvanov]

Did you manage to fix this? I have the same issue.

[MichaelOpal]

Same here. This is marked as resolved without any sort of resolution. What fixed it?

[rumexcrisp]

check if the configured machine learning url in admin settings is reachable by immich-server

[JaredTamana]

Also struggling with this. Tried localhost, local IP, public IP, and the default. I note that no port is assigned to immich-machine-learning when examining from Docker Desktop, not sure if this is as intended, but given the missing IP in the log output I'm going to assume this is a misconfiguration: Listening at: http://[::]:3003 (8)

immich-machine-learning

[06/03/25 23:29:41] INFO     Starting gunicorn 23.0.0                           

[06/03/25 23:29:41] INFO     Listening at: http://[::]:3003 (8)                 

[06/03/25 23:29:41] INFO     Using worker: immich_ml.config.CustomUvicornWorker 

[06/03/25 23:29:41] INFO     Booting worker with pid: 9                         

[06/03/25 23:29:42] INFO     Started server process [9]                         

[06/03/25 23:29:42] INFO     Waiting for application startup.                   

[06/03/25 23:29:42] INFO     Created in-memory cache with unloading after 300s  

                             of inactivity.                                     

[06/03/25 23:29:42] INFO     Initialized request thread pool with 12 threads.   

[06/03/25 23:29:42] INFO     Application startup complete.                      
immich-server          

[Nest] 19  - 06/03/2025, 11:29:52 PM    WARN [Api:MachineLearningRepository~qrlch3le] Machine learning request to "http://192.168.0.129:3003" failed: fetch failed

[Nest] 19  - 06/03/2025, 11:29:52 PM    WARN [Api:MachineLearningRepository~qrlch3le] Machine learning request to "http://127.0.0.1:3003" failed: fetch failed

[Nest] 19  - 06/03/2025, 11:29:52 PM   ERROR [Api:ErrorInterceptor~qrlch3le] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-L-16-SigLIP2-256__webli","options":{"language":"en-US"}}}}' failed for all URLs

Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-L-16-SigLIP2-256__webli","options":{"language":"en-US"}}}}' failed for all URLs

    at MachineLearningRepository.predict (/usr/src/app/dist/repositories/machine-learning.repository.js:98:15)

    at process.processTicksAndRejections (node:internal/process/task_queues:105:5)

    at async MachineLearningRepository.encodeText (/usr/src/app/dist/repositories/machine-learning.repository.js:121:26)

    at async SearchService.searchSmart (/usr/src/app/dist/services/search.service.js:62:27)

[Ptitet]

I have the same problem. The immich server is hosted on my NAS, and the remote ml server is on my computer. The remote ml is up, I can access it in the browser from any machine on my local network (going to the root url returns {"message": "Immich ML"}), and I can also curl it from the NAS, but the immich server can't access it (ERROR: unable to run job handler : error : machine learning request failed for all urls).

[cberthe067]

Same here too ... Did not manage to solve it ... It's a nasty bug ...

[oreze]

Same here, I host it using lxc (proxmox-scripts). Works with default model, doesn't work with ViT-SO400M-16-SigLIP2-384__webli. Here's part of my logs:

[Nest] 339  - 11/06/2025, 7:33:39 PM   DEBUG [Api:LoggingInterceptor~eatx1cre] GET /api/assets/98d33c38-9497-4498-8c4e-829f7ac5f165/thumbnail?size=thumbnail&c=5igKHITmd2eLeYePdXeYcIYJZw%3D%3D 200 47.52ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:33:39 PM   DEBUG [Api:LoggingInterceptor~93v955kc] GET /api/assets/14aab7e0-4b5b-4653-801f-fb755ab8e15b/thumbnail?size=thumbnail&c=pRgOFAKl%2BIm3Z4tGeHfHoIELGg%3D%3D 200 42.32ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:33:39 PM   DEBUG [Api:LoggingInterceptor~8bi0bnrw] GET /api/assets/d357e5f3-0283-44f7-a54c-8df69d2975d1/thumbnail?size=thumbnail&c=UxgOFAJnB4eIiXl3hnhFgGIzBw%3D%3D 200 25.29ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:33:45 PM    WARN [Api:MachineLearningRepository~jp3w1zns] Machine learning request to "http://127.0.0.1:3003" failed: fetch failed
[Nest] 339  - 11/06/2025, 7:33:45 PM     LOG [Api:MachineLearningRepository~jp3w1zns] Machine learning server became unhealthy (http://127.0.0.1:3003).
[Nest] 339  - 11/06/2025, 7:33:45 PM   ERROR [Api:ErrorInterceptor~jp3w1zns] Unknown error: Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-SO400M-16-SigLIP2-384__webli","options":{"language":"en-US"}}}}' failed for all URLs
Error: Machine learning request '{"clip":{"textual":{"modelName":"ViT-SO400M-16-SigLIP2-384__webli","options":{"language":"en-US"}}}}' failed for all URLs
    at MachineLearningRepository.predict (/opt/immich/app/dist/repositories/machine-learning.repository.js:117:15)
    at async MachineLearningRepository.encodeText (/opt/immich/app/dist/repositories/machine-learning.repository.js:140:26)
    at async SearchService.searchSmart (/opt/immich/app/dist/services/search.service.js:94:29)
[Nest] 339  - 11/06/2025, 7:33:45 PM   DEBUG [Api:LoggingInterceptor~jp3w1zns] POST /api/search/smart 200 5349.57ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:33:45 PM   DEBUG [Api:GlobalExceptionFilter~jp3w1zns] HttpException(500): {"message":"Failed to search smart","error":"Internal Server Error","statusCode":500}
[Nest] 93  - 11/06/2025, 7:33:45 PM    WARN [Microservices:MachineLearningRepository] Machine learning request to "http://127.0.0.1:3003" failed: fetch failed
[Nest] 93  - 11/06/2025, 7:33:45 PM     LOG [Microservices:MachineLearningRepository] Machine learning server became unhealthy (http://127.0.0.1:3003).
[Nest] 93  - 11/06/2025, 7:33:45 PM   ERROR [Microservices:{"id":"09981726-5701-42a9-9897-0aafb37d4221"}] Unable to run job handler (SmartSearch): Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-SO400M-16-SigLIP2-384__webli"}}}' failed for all URLs
Error: Machine learning request '{"clip":{"visual":{"modelName":"ViT-SO400M-16-SigLIP2-384__webli"}}}' failed for all URLs
    at MachineLearningRepository.predict (/opt/immich/app/dist/repositories/machine-learning.repository.js:117:15)
    at async MachineLearningRepository.encodeImage (/opt/immich/app/dist/repositories/machine-learning.repository.js:135:26)
    at async SmartInfoService.handleEncodeClip (/opt/immich/app/dist/services/smart-info.service.js:91:27)
    at async JobService.onJobRun (/opt/immich/app/dist/services/job.service.js:199:30)
    at async EventRepository.onEvent (/opt/immich/app/dist/repositories/event.repository.js:91:13)
    at async /opt/immich/app/node_modules/.pnpm/bullmq@5.61.2/node_modules/bullmq/dist/cjs/classes/worker.js:528:32
[Nest] 93  - 11/06/2025, 7:34:01 PM     LOG [Microservices:MachineLearningRepository] Machine learning server became healthy (http://127.0.0.1:3003).
[Nest] 339  - 11/06/2025, 7:34:01 PM     LOG [Api:MachineLearningRepository~rdt4pkfc] Machine learning server became healthy (http://127.0.0.1:3003).
[Nest] 339  - 11/06/2025, 7:39:12 PM   DEBUG [Api:LoggingInterceptor~uc9kitqa] GET /api/notifications?unread=true 200 38.21ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:39:12 PM   DEBUG [Api:LoggingInterceptor~9sp713e9] GET /api/timeline/buckets?visibility=timeline&withPartners=true&withStacked=true 200 115.58ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:39:13 PM   DEBUG [Api:LoggingInterceptor~5el7rtfu] GET /api/timeline/bucket?timeBucket=2025-11-01T00%3A00%3A00.000Z&visibility=timeline&withPartners=true&withStacked=true 200 71.93ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:39:14 PM   DEBUG [Api:LoggingInterceptor~ac887j8f] GET /api/server/storage 200 0.43ms ::ffff:192.168.0.50
[Nest] 339  - 11/06/2025, 7:39:14 PM   DEBUG [Api:LoggingInterceptor~ywy60u5r] GET /api/admin/users?withDeleted=true 200 16.34ms ::ffff:192.168.0.50

Also, ML service crashes frequently despite having enough (i think) resources - 3 N100 CPUs, 8Gb RAM and 8Gb SWAP

[jeremyy294]

Exactly similar setup - ML is hosted on a separate LXC which is accessible via curl without any issue, but smart search simply does not work. The only way I can make smart search is to leave the ML server as the default "http://immich-machine-learning:3003/ and use the ML running on the primary LXC.

[dajeffers]

Running this in Kubernetes, the machine learning container logs showed an inability to write to the volume. Once I fixed volume permissions, it worked fine.