[BUG] Unable to download 'buffalo_l', 'ViT-B-32__openai' and 'microsoft/resnet-50' so machine learning functions are not working

[acios]

The bug

When I try to search, using web or app, it does not give any result, checking the log of docker immich_machine_learning, it gives the following log, shows that the models needed was not downloaded, seems it does not even have the permission to make the folder. I try to manually make the folders, the log changes a bit (from folder not exist to unable to download)

File "/usr/src/app/main.py", line 121, in load
await loop.run_in_executor(app.state.thread_pool, _load)
File "/usr/local/lib/python3.11/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/src/app/main.py", line 101, in _load
model.load()
File "/usr/src/app/models/base.py", line 65, in load
self.download()
File "/usr/src/app/models/base.py", line 60, in download
self._download()
File "/usr/src/app/models/base.py", line 84, in _download
snapshot_download(
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/_snapshot_download.py", line 189, in snapshot_download
repo_info = api.repo_info(repo_id=repo_id, repo_type=repo_type, revision=revision, token=token)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1888, in repo_info
return method(
^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_validators.py", line 118, in _inner_fn
return fn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/hf_api.py", line 1697, in model_info
r = get_session().get(path, headers=headers, timeout=timeout, params=params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 602, in get
return self.request("GET", url, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 589, in request
resp = self.send(prep, **send_kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/requests/sessions.py", line 703, in send
r = adapter.send(request, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/huggingface_hub/utils/_http.py", line 63, in send
return super().send(request, *args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/venv/lib/python3.11/site-packages/requests/adapters.py", line 519, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/ViT-B-32__openai/revision/main (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f57aeb8c390>: Failed to establish a new connection: [Errno 111] Connection refused'))"), '(Request ID: 184bad77-76e5-48ba-862b-5cf24aef50b8)')
[11/19/23 00:34:19] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:34:19] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:34:19] INFO Cleared cache directory for model
'ViT-B-32__openai'.
[11/19/23 00:34:19] INFO Downloading image classification model
'microsoft/resnet-50'.This may take a while.
[11/19/23 00:34:19] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:34:19] INFO Cleared cache directory for model
'ViT-B-32__openai'.
[11/19/23 00:34:19] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:34:19] WARNING Failed to load image-classification model
'microsoft/resnet-50'.Clearing cache and retrying.
[11/19/23 00:34:19] WARNING Attempted to clear cache for model
'microsoft/resnet-50' but cache directory does not
exist.
[11/19/23 00:36:29] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:36:29] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:36:29] WARNING Attempted to clear cache for model
'ViT-B-32__openai' but cache directory does not
exist.
[11/19/23 00:38:40] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:38:40] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:38:40] WARNING Attempted to clear cache for model
'ViT-B-32__openai' but cache directory does not
exist.
[11/19/23 00:40:51] INFO Downloading facial recognition model
'buffalo_l'.This may take a while.
[11/19/23 00:40:51] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:40:51] WARNING Attempted to clear cache for model
'ViT-B-32__openai' but cache directory does not
exist.
[11/19/23 00:43:02] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:43:02] WARNING Failed to load facial-recognition model
'buffalo_l'.Clearing cache and retrying.
[11/19/23 00:43:02] WARNING Attempted to clear cache for model 'buffalo_l' but
cache directory does not exist.
[11/19/23 00:45:13] INFO Downloading clip model 'ViT-B-32__openai'.This may
take a while.
[11/19/23 00:45:13] WARNING Failed to load clip model
'ViT-B-32__openai'.Clearing cache and retrying.
[11/19/23 00:45:13] WARNING Attempted to clear cache for model
'ViT-B-32__openai' but cache directory does not
exist.

The OS that Immich Server is running on

Debian GNU/Linux 11 (bullseye)

Version of Immich Server

v1.87.0

Version of Immich Mobile App

v1.87.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

GNU nano 5.4                                                                                                    docker-compose.yml *                                                                                                            
version: "3.8"

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    extends:
       file: hwaccel.yml
       service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:3995fe6ea6a619313e31046bd3c8643f9e70f8f2b294ff82659d409b47d06abb
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:874f566dd512d79cf74f59754833e869ae76ece96716d153b0fa3e64aec88d92
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

GNU nano 5.4                                                                                                           .env                                                                                                                     
# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/truenas_nfs/Photos

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=vowansodiujvlc
DB_PASSWORD=postgres

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

Reproduction steps

1.compose the dockers
2.upload some photos
3.try to search for anything
4.no result
...

Additional information

I was trying to follow this issue:
#4117

to download the files manually but the link has expired

[aviv926]

I would like you to check what your permissions are in the
model-cache:/cache

To do this you can use the command ls -l in the folder and upload a screenshot here
I'm not an Immich expert but I'll try to help as much as I can

[aviv926]

Check if permissions are shown to you properly as they appear to me
The permissions are displayed on the right as wxr

[acios]

I only have 1 file in the cache folder:
-rw-r--r-- 1 root root 1 Nov 19 01:20 version.txt

[aviv926]

I only have 1 file in the cache folder: -rw-r--r-- 1 root root 1 Nov 19 01:20 version.txt

Pay attention to do the ls -l command on the app folder so that you can see the permissions on the "cache" folder

[acios]

this is in the app folder, which cache is not here

this is in the root folder where cache is located:

I'm doing this inside the docker I hope it is the right way?

Thanks for the help btw

[aviv926]

this is in the app folder, which cache is not here

this is in the root folder where cache is located:

I'm doing this inside the docker I hope it is the right way?

Thanks for the help btw

There doesn't seem to be a problem with folder permissions
Can you take a picture of your settings that are on the page
/admin/system-settings

[aviv926]

Also there seems to be a blockage that I had in the past
As shown in the log you attached above:

requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/ViT-B-32__openai/revision/main (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f57aeb8c390>: Failed to establish a new connection:

I would recommend you to try using a VPN like proton vpn to bypass this block (it worked for me)

[acios]

this is in the app folder, which cache is not here
this is in the root folder where cache is located:
I'm doing this inside the docker I hope it is the right way?
Thanks for the help btw

There doesn't seem to be a problem with folder permissions Can you take a picture of your settings that are on the page /admin/system-settings

mine is exactly the same as your screenshot

[acios]

Also there seems to be a blockage that I had in the past As shown in the log you attached above:

requests.exceptions.ConnectionError: (MaxRetryError("HTTPSConnectionPool(host='huggingface.co', port=443): Max retries exceeded with url: /api/models/immich-app/ViT-B-32__openai/revision/main (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f57aeb8c390>: Failed to establish a new connection:

I would recommend you to try using a VPN like proton vpn to bypass this block (it worked for me)

I'm already using proxy through my side router which is running passwall tunneling my internet traffic through a remote server. I'll try the VPN solution, maybe my remote server is blocked.