[Unraid] Hourly version check wakes up both data disk and parity disk

[nedevski]

I have searched the existing issues, both open and closed, to make sure this is not a duplicate report.

• Yes

The bug

I am hosting Immich with Unraid. It's a simple setup:

• Database is stored on SSD
• Immich data is stored on the Unraid array with a SSD Cache enabled, pinned to only 1 HDD

However I noticed every hour two disks wake up - Data disk and Parity disk (meaning Immich triggers both reads and writes on the array). After two days of debugging, I narrowed it down to the version check.

Disabling the version check from the Settings prevents waking up the HDDs.

My question is - why would a version check (especially if no new version has been found) would need reading or writing to the UPLOAD_LOCATION?

I am trying to keep all of my disks spun down, since my server is idle 95% of the time. It would be nice if the version check logic is changed in such a way that it doesn't wake up any disks.
I guess it's fine to update a file if there IS a new version, but definitely not if there isn't anything new.

The OS that Immich Server is running on

Unraid 7.1.4

Version of Immich Server

v2.1.0

Version of Immich Mobile App

latest

Platform with the issue

• Server
• Web
• Mobile

Device make and model

No response

Your docker-compose.yml content

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.transcoding.yml
    #   service: cpu # set to one of [nvenc, quicksync, rkmpp, vaapi, vaapi-wsl] for accelerated transcoding
    volumes:
      # Do not edit the next line. If you want to change the media storage location on your system, edit the value of UPLOAD_LOCATION in the .env file
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - '2283:2283'
    depends_on:
      - redis
      - database
    restart: always
    healthcheck:
      disable: false

  immich-machine-learning:
    container_name: immich_machine_learning
    # For hardware acceleration, add one of -[armnn, cuda, openvino] to the image tag.
    # Example tag: ${IMMICH_VERSION:-release}-cuda
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    # extends: # uncomment this section for hardware acceleration - see https://immich.app/docs/features/ml-hardware-acceleration
    #   file: hwaccel.ml.yml
    #   service: cpu # set to one of [armnn, cuda, openvino, openvino-wsl] for accelerated inference - use the `-wsl` version for WSL2 where applicable
    volumes:
      - model-cache:/cache
    env_file:
      - .env
    restart: always
    healthcheck:
      disable: true

  redis:
    container_name: immich_redis
    image: docker.io/redis:6.2-alpine@sha256:148bb5411c184abd288d9aaed139c98123eeb8824c5d3fce03cf721db58066d8
    healthcheck:
      test: redis-cli ping || exit 1
    restart: always

  database:
    container_name: immich_postgres
    image: docker.io/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:739cdd626151ff1f796dc95a6591b55a714f341c737e27f045019ceabf8e8c52
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      # Do not edit the next line. If you want to change the database storage location on your system, edit the value of DB_DATA_LOCATION in the .env file
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    healthcheck:
      test: >-
        pg_isready --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" || exit 1;
        Chksum="$$(psql --dbname="$${POSTGRES_DB}" --username="$${POSTGRES_USER}" --tuples-only --no-align
        --command='SELECT COALESCE(SUM(checksum_failures), 0) FROM pg_stat_database')";
        echo "checksum failure count is $$Chksum";
        [ "$$Chksum" = '0' ] || exit 1
      interval: 5m
    command: >-
      postgres
      -c shared_preload_libraries=vectors.so
      -c 'search_path="$$user", public, vectors'
      -c logging_collector=on
      -c max_wal_size=2GB
      -c shared_buffers=512MB
      -c wal_compression=on
    restart: always

volumes:
  model-cache:

Your .env content

# The location where your uploaded files are stored
UPLOAD_LOCATION=/mnt/user/immich/library
# The location where your database files are stored
DB_DATA_LOCATION=/mnt/user/docker/appdata/immich/postgres

# To set a timezone, uncomment the next line and change Etc/UTC to a TZ identifier from this list: https://en.wikipedia.org/wiki/List_of_tz_database_time_zones#List
TZ=Europe/Sofia

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
IMMICH_VERSION=release

# Connection secret for postgres. You should change it to a random password
# Please use only the characters `A-Za-z0-9`, without special characters or spaces
DB_PASSWORD=redacted

# The values below this line do not need to be changed
###################################################################################
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

Reproduction steps

1. Install Immich
   1.1. Postgres on a SSD Cache share
   1.2. Library on an Array share. Configure the share to use only 1 hard drive.
   1.3. Enable DEBUG logs
2. Set HDD spin down delay to 15 minutes
3. Don't use the app, don't open the Web UI for more than an hour
4. Observe both the Parity drive and Array drive wake up
5. Check logs - drives woke up exactly at the version check time
6. Disable version check
7. Wait at least an hour and observe logs - version check still runs, but now doesn't wake up HDDs

Relevant log output

DEBUG [Microservices:VersionService] Running version check

Additional information

No response

[bo0tzz]

why would a version check (especially if no new version has been found) would need reading or writing to the UPLOAD_LOCATION?

It doesn't, so I'm not sure what's happening there. Are you sure your postgres dir isn't going to the disks after all? Could it be that your redis instance is configured to write to them?

[nedevski]

It doesn't, so I'm not sure what's happening there. Are you sure your postgres dir isn't going to the disks after all? Could it be that your redis instance is configured to write to them?

Yes, Postgres instance is in my docker share, which is Cache only. For redis I'm using the default volume, since its cache anyway, and that should be on the cache drive as well.

No file activity logged in Unraid either during wake up hours. No additional logs logged either (using Verbose log levels)

[skatsubo]

If it's possible to install fatrace on Unraid then could you run it on the Array share (HDDs)?

1. cd /path/to/mount
2. fatrace -t -c
3. Re-enable version check and wait.

[nedevski]

Sorry, but I wasn't able to find any information on how to do that. Searched in apps, Google suggested "Nerd pack" and "Dev pack" which seem to not be compatible with Unraid 7+. If you point me in the right direction I will be more than happy to do it.

That being said - I have the "File Activity" plugin which is supposed to do the same - I can see all file movements on all shares, including thumb and preview generations, opens, updates, etc.

That plugin did not register any file activity during wakeups.

[skatsubo]

Re: fatrace.
I was able to install and run it from command line (on a freshly created Unraid qemu/kvm without license)

cd /tmp
wget https://slackware.halpanet.org/slackdce/packages/15.0/x86_64/system/fatrace/fatrace-0.19.1-x86_64-1_slackdce.txz
upgradepkg --install-new fatrace-0.19.1-x86_64-1_slackdce.txz

root@unraid:/tmp# fatrace -t
    fatrace: Failed to add watch for /usr: Operation not supported
    fatrace: Failed to add watch for /lib: Operation not supported
    01:43:51.578219 sh(37262): RO  /bin/bash
    01:43:51.578219 sh(37262): RO  /lib64/ld-linux-x86-64-2.42.so
    01:43:51.582243 udevadm(37262): RO  /lib64/ld-linux-x86-64-2.42.so
    01:43:51.582942 udevadm(37262): C   /bin/bash
    01:43:51.616396 emhttpd(1990): RCO /boot/config
    01:43:51.616396 emhttpd(1990): RCO /boot
    01:43:51.616396 emhttpd(1990): +D  /var/local/emhttp

Also I've seen un-get https://github.com/ich777/un-get but haven't tried it.

Re:

the "File Activity" plugin which is supposed to do the same <...>
That plugin did not register any file activity during wakeups.

Interesting.
So we can cross-validate File Activity with fatrace: it uses fanotify instead of inotify.

[nedevski]

Installed and ran it. It produces an enormous amount of logs.

• I tried using | grep "/mnt/user/immich but then it still printed a lot of:
  fatrace: failed to open /proc/4016331: No such file or directory
• I tried saving logs to the filesystem but for around 10-20 seconds it produced a 100MB text file.. Maybe if I time it right I can catch something, but in that many logs I'm not sure it will be valid.

On another note - Those same 2 drives still spin up occasionally, although not that often, like 1-2 times per day. Sometimes it's only the data drive, indicating only reading.

Keep in mind that I have a cache drive for Immich, in theory all writes should first go to that drive?

I'm unsure on how to proceed debugging this, but if you have any suggestions - I will do it.

[skatsubo]

Re: fatrace.

Is it still producing a lot when running from the mount / drive and with -c option?

cd /mnt/user/immich
fatrace -t -c

Those not grepped messages go through stderr; you can redirect them to stdout to be grepped too. Smth like fatrace -t -c 2>&1 | grep immich

[nedevski]

fatrace -t -c resulted in fatrace: Failed to add watch for .: No such device when in /mnt/user/immich
If I don't use the -c, the 2nd command produces a much more manageable log and I found this:

Context - All my data is on the immich share, which is pinned to disk2 (with Cache enabled + mover once per day)
All my persistent container data (including docker.img) is in the docker share, which is set to use only the Cache SSD.

Now, I have no idea why the DB files ended up on disk2 but seems like that's the underlying issue?

So as for this ticket - I guess it's a false alarm/misconfiguration on my part.
Now I just need to see how to "merge" the files from both disks and dump them in the Cache only.

Thanks a lot for your help in debugging this, much appreciated!