When changing thumbnail folder, it defaults back to the previous folder a few hours after changing

[aaronjrodrigues]

The bug

I followed this guide (https://immich.app/docs/guides/custom-locations/) to make thumbnails get stored on my ssd while my uploads and everything else get stored on my storage box. However I've noticed that it works fine for a few hours and then it reverts to storing the thumbnails on the storage box again.

I'm sure this is happening because I delete the thumbnail folder on the storage box multiple times but it comes back and it starts generating images in the old location again.

The OS that Immich Server is running on

Ubuntu 22.04, ARM64

Version of Immich Server

v1.112.1

Version of Immich Mobile App

v1.112.1

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#
​
name: immich
​
services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false
​
  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: false
​
  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:e3b17ba9479deec4b7d1eeec1548a253acc5374d68d3b27937fcfe4df8d18c7e
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always
​
  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/home/ubuntu/gringotts/immich
THUMB_LOCATION=/home/ubuntu/immich/thumbs
# The location where your database files are stored
DB_DATA_LOCATION=/home/ubuntu/immich/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
# TZ=Etc/UTC

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=redacted

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Setup immich server using this guide (https://immich.app/docs/install/docker-compose). In this setup, assume by default the upload location is /library and as a result thumbs, uploads, etc will have the path have the path /libary/thumbs, /library/uploads.

2. Follow this guide https://immich.app/docs/guides/custom-locations/ and create a custom location for thumbnails ONLY. So now, assume we've set the custom location for thumbs to be /ssd/thumbs.

3. Delete the old thumbs folder at /library/thumbs and regenerate all thumbnails, so all new thumbnails will be created at /ssd/thumbs.

4. Wait for a few days and notice how Immich will revert to storing thumbs in the default location (/libary/thumbs) and all previously generated thumbnails for images wont be able to be viewed on immich web as it changed the thumbnail path back to the default one.

If you restart immich, it'll go back to using the new thumbnail folder however after a few hours it'll go back to the old thumbnail folder.

I've attached logs under 'Relevant log output' that shows the errors that Immich outputs when trying to view thumbnails after immich changed the thumbnail location back to the old one.

Relevant log output

[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:LoggerRepository~rzicjm2h] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/53/50/5350e04c-0b79-419d-ad82-ccf69e9ecdba-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM    WARN [Api:ExpressAdapter~rzicjm2h] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:ExceptionsHandler~rzicjm2h] ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/53/50/5350e04c-0b79-419d-ad82-ccf69e9ecdba-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/53/50/5350e04c-0b79-419d-ad82-ccf69e9ecdba-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:LoggerRepository~6sc8jmxy] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/c5/68/c568c02f-d58d-4f8a-ac38-dd98f20b7a58-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM    WARN [Api:ExpressAdapter~6sc8jmxy] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:ExceptionsHandler~6sc8jmxy] ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/c5/68/c568c02f-d58d-4f8a-ac38-dd98f20b7a58-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/c5/68/c568c02f-d58d-4f8a-ac38-dd98f20b7a58-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:LoggerRepository~37faqe7q] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/26/61/2661e9df-7aec-42c5-ac2e-04fcfe9836bc-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM    WARN [Api:ExpressAdapter~37faqe7q] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:ExceptionsHandler~37faqe7q] ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/26/61/2661e9df-7aec-42c5-ac2e-04fcfe9836bc-thumbnail.webp'
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/26/61/2661e9df-7aec-42c5-ac2e-04fcfe9836bc-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:LoggerRepository~nzebxe8f] Unable to send file: Error
Error: ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/13/cc/13cc06fa-9dc5-45e3-95b6-705b5fa835b8-thumbnail.webp'
    at async access (node:internal/fs/promises:606:10)
    at async sendFile (/usr/src/app/dist/utils/file.js:55:9)
    at async AssetMediaController.viewAsset (/usr/src/app/dist/controllers/asset-media.controller.js:57:9)
[Nest] 18  - 08/20/2024, 12:42:42 PM    WARN [Api:ExpressAdapter~nzebxe8f] Content-Type doesn't match Reply body, you might need a custom ExceptionFilter for non-JSON responses
[Nest] 18  - 08/20/2024, 12:42:42 PM   ERROR [Api:ExceptionsHandler~nzebxe8f] ENOENT: no such file or directory, access 'upload/thumbs/496d4685-9669-437e-a2e4-25324663b100/13/cc/13cc06fa-9dc5-45e3-95b6-705b5fa835b8-thumbnail.webp'

Additional information

No response

[alextran1502]

Can you double-check if the drive location at THUMB_LOCATION is mounted correctly? The log you are seeing is the internal docker path, which is correct

[aaronjrodrigues]

Yep, it's mounted correctly. It's a very odd issue, not sure what it could be.

[aaronjrodrigues]

@alextran1502

I had a look at the compose file and I have a suspicion as to what could be happening

      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
      - /etc/localtime:/etc/localtime:ro

The guide it told me to add the following line "- ${THUMB_LOCATION}:/usr/src/app/upload/thumbs".

So what I think the issue is that since the upload folder (usr/src/app/upload) mounts to /home/ubuntu/gringotts and thumbnails (usr/src/app/upload/thumbs) mounts to /home/ubuntu/immich/thumbs. Immich somehow gets confused and treats /home/ubuntu/gringotts/immich/thumbs as the thumbnail folder instead of /home/ubuntu/immich/thumbs because usr/src/app/upload mounts to /home/ubuntu/gringotts/immich.

I hope I made sense, it was so confusing even for me typing it down 😭

[thrhgh]

I read through the post because I run into exactly the same issue. I assume the issue is due to the same path of the mount as @aaronjrodrigues already assumed. Anyone found a fix for this?

[Kaiohshin]

My Immich instance is also refusing to use the thumbnail path I set. Looking forward to a solution

[aaronjrodrigues]

@Kaiohshin clean install as in you deleted the database?

[Kaiohshin]

yes, I deleted all my volumes.

[Kaiohshin]

This is what I used for both server and microservices
volumes:
- ${UPLOAD_LOCATION}:/usr/src/app/upload/upload
- ${THUMBS_LOCATION}:/usr/src/app/upload/thumbs
- ${VIDEO_LOCATION}:/usr/src/app/upload/encoded-video
- ${PROFILE_LOCATION}:/usr/src/app/upload/profile
- ${LIBRARY_LOCATION}:/usr/src/app/upload/library
- /etc/localtime:/etc/localtime:ro
- /Memories:/usr/src/app/external:ro

[aaronjrodrigues]

May I please see the hierarchy of your Immich uploads folder? Because I want to know why your uploads folder is ${UPLOAD_LOCATION}:/usr/src/app/upload/upload instead of ${UPLOAD_LOCATION}:/usr/src/app/upload/

[Kaiohshin]

[aaronjrodrigues]

@alextran1502 Reinstalled immich on a new system with another hetzner storage box mounted, set up the same configuration (locally storing thumbnails on ssd while the rest stored on storage box) and the issue came up again. I don't think it's an issue with the mount itself but a bug in immich. Also updating to the latest version didn't fix the issue.

[bo0tzz]

These are docker mount points that Immich isn't even aware of, I think it's pretty much impossible for it to be a bug in Immich.

[aaronjrodrigues]

These are docker mount points that Immich isn't even aware of, I think it's pretty much impossible for it to be a bug in Immich.

@bo0tzz I understand what you mean, but i have a suspicion because since in the compose file listed below, /usr/src/app/upload points to my storage box (for my uploads) and usr/src/app/uploads/thumbs mounts to my local filesystem (for thumbs). I have a feeling since technically /thumbs is a subfolder of usr/src/app/uploads and since usr/src/app/uploads actually mounts to my storage box, immich just stores my thumbs in the storage box.

I hope I made sense as its kinda confusing to type it out, below is part of the docker-compose file that im referring to.

volumes: # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file - ${UPLOAD_LOCATION}:/usr/src/app/upload - ${THUMB_LOCATION}:/usr/src/app/upload/thumbs - /etc/localtime:/etc/localtime:ro

[thrhgh]

I still have this issue. Did anyone found a solution to this? @aaronjrodrigues

[lukee7]

I ran into the same issue right now.

I wanted the uploads to land on the HDD, but the thumbs on the SSD.

.env

UPLOAD_LOCATION=/srv/dev-disk-by-uuid-b7c7c712-68de-4052-ae40-d548a887932f/fotos/
THUMB_LOCATION=/opt/immich-app/thumbnail/

.docker-compose

- ${UPLOAD_LOCATION}:/usr/src/app/upload
- ${THUMB_LOCATION}:/usr/src/app/upload/thumbs
- /srv/dev-disk-by-uuid-b7c7c712-68de-4052-ae40-d548a887932f/fotos:/mnt/fotos:ro  // used as external library

As the thumbs were created on the HDD (which is used as external library too) immich started creating thumbnails for the thumbnails in an endless loop 😨 👻

I guess the warning box here https://immich.app/docs/guides/custom-locations explains our problem!?!

Because of the underlying properties of docker bind mounts, it is not recommended to mount the upload/ and library/ folders as separate bind mounts if they are on the same device. For this reason, we mount the HDD or network storage to /usr/src/app/upload and then mount the folders we want quick access to below this folder.

Seems to be a problem with docker bind mounts in a folder/subfolder structure as immich is using internally?

[bo0tzz]

Somehow didn't think of it when first dealing with this thread, but y'all should make sure you don't still have an old orphaned microservices container sitting around.

[thrhgh]

@bo0tzz Thanks for your advice, deleting the old containers did not help to fix this issue, still persisting for me.

[IceBoosteR]

Same issue here.
First of, the file structure in Immich is a bit stupid I think.
We have the default location inside the main docker container /usr/src/app/upload. Well I would think this is the location where you would actually upload the files to. But it actually has these folders below

root@d73de470be95:/usr/src/app/upload# ls
backups  encoded-video  library  profile  thumbs  upload

Thinking about that, the default config in docker compose tells docker, that in the root directory where you execute docker compose up -d the ./library is located - and this is where Immich would expect to store all these files. Now if you want (like I do) to put the thumbs and the encoded-video folder on a different device/mount/whatever, you would mount it basically in a an unclean way (will work depending on how you do it, but nvmd). So what I would like to see is a better structure here. Usually I do mount my external shares, NFS etc. in /mnt and direct the docker engine to mount the volume from this location. Therefore I have it clean and in the structure that I need it.
But with however method I tried it, Immich was unable to start with the errors you guys saw aswell. Even if it actually worked (just go into a shell in that docker and see if the mount works or not...). So this is why I applied IMMICH_IGNORE_MOUNT_CHECK_ERRORS=true in the env file to get rid of the error.
What I have done now is to mount my external shares directly through /etc/fstab into the location Immich wants, so docker and the container have nothing to do with it.
As an example /docker/Immich/library/thumbs where this is basically a cifs mount.
I can confirm that this worked for me, all thumbs are store on my NAS where I want it to be.

TL;DR I solved the issue with linux OS stuff, but it is basically an actual error within Immich on the check-script I guess! And the folder strcuture should be much cleaner for a better way to use external mounts

[aaronjrodrigues]

@mmomjian may you please let me know which pr/commit fixed the issue? Wanted to have a look at what actually was the underlying issues for this.

[mmomjian]

Edit: I see I closed it. This isn’t an immich bug, and never was, so there’s no PR

[aaronjrodrigues]

@mmomjian so technically the issue still exists right? Got confused because the reason for the thread being closed was that the issue got resolved.

[mmomjian]

If there is an issue here, which I’m not totally convinced by, it would be a docker issue. The mount point structure is transparent to immich, we don’t have the ability to “choose” which mount we write to. Docker / overlay FS handles all that

[aaronjrodrigues]

I got an email stating that you closed this thread as resolved. Hence why I had tagged, apologies.

…

On Wed, Jan 22, 2025 at 12:49, Matthew Momjian ***@***.***(mailto:On Wed, Jan 22, 2025 at 12:49, Matthew Momjian <<a href=)> wrote: Why did you tag me in this? Have I ever participated in this thread..? — Reply to this email directly, [view it on GitHub](#11944 (reply in thread)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/AJDRWEOWKSUI5NOGMADX7KD2L5ZVLAVCNFSM6AAAAABM3EZ3E2VHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTCOJRGQ4DQOI). You are receiving this because you were mentioned.Message ID: ***@***.***>

[mmomjian]

I edited my reply, sorry. But this was never an immich bug, so there’s no PR

[floriankilian]

Linking this to a more recent discussion with the same symptom (#24314) — seeing the exact same behavior described here: thumbnails silently fall back to the default location, with no mount changes and no restart in between. I posted timestamps and docker inspect output over there that pin down the exact moment the switch happens: #24314

Might be worth linking the two threads together in case they're the same underlying issue.