Scan & offline assets issue - old compose file with Database

[matticas]

Hi, this issue has been since ~1.100? I think. I started Immich around 1.9x ? I am now on 1.105.1

I think the issue is with my old Portainer-Stack file settings??

Symptoms are;

1. All my External Library Assets are "offline".
2. Scan, Forced Re-scan all do nothing on external library. I get 2 seconds of CPU usage, then nothing, and no changes.

I have checked read-only access within both containers and they work with the external libraries;

• immich-server: upload/
• immich-server: /My_Photos/
• immich-microservices: upload/
• immich-microservices: /My_Photos/

All four of those photos have read-only access.

I have checked the external paths, and validated them successfully.

The rest of Immich works great, including new images from 2x Android phones nightly, including facial recognition etc.

But, I am unsure how to update the PG database from docker volume to a mounted volume? ( I don't want to loose all my albums and face tagging for 20yrs of images).

My Settings;

Portainer Stack;

version: "3.8"

#
# WARNING: Make sure to use the docker-compose.yml of the current release:
#
# https://github.com/immich-app/immich/releases/latest/download/docker-compose.yml
#
# The compose file on main may not be compatible with the latest release.
#

name: immich

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: [ "start.sh", "immich" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/Shed4/My Photos/:/My_Photos:ro
    env_file:
      - stack.env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: [ "start.sh", "microservices" ]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/localtime:/etc/localtime:ro
      - /media/Shed4/My Photos/:/My_Photos:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:c5a607fb6e1bb15d32bbcf14db22787d19e428d59e31a5da67511b49bb0f1ccc
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

volumes:
  pgdata:
  model-cache:

The latest Portainer Stack; (image paths, and namely the volume for database container)

  redis:
    container_name: immich_redis
    image: registry.hub.docker.com/library/redis:6.2-alpine@sha256:84882e87b54734154586e5f8abd4dce69fe7311315e2fc6d67c29614c8de2672
    restart: always

  database:
    container_name: immich_postgres
    image: registry.hub.docker.com/tensorchord/pgvecto-rs:pg14-v0.2.0@sha256:90724186f0a3517cf6914295b5ab410db9ce23190a2d9d0b9dd6463e3fa298f0
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
      POSTGRES_INITDB_ARGS: '--data-checksums'
    volumes:
      - ${DB_DATA_LOCATION}:/var/lib/postgresql/data
    restart: always
    command: ["postgres", "-c" ,"shared_preload_libraries=vectors.so", "-c", 'search_path="$$user", public, vectors', "-c", "logging_collector=on", "-c", "max_wal_size=2GB", "-c", "shared_buffers=512MB", "-c", "wal_compression=on"]

Would this be causing my scan and offline issues?

How do I migrate the docker volume to my mounted disk volume?

Thanks!

[alextran1502]

All my External Library Assets are "offline".

Can you check if the mount is actually there?

[matticas]

Hi Alex,

Firstly, thanks for Immich!

This is how I check; (correct me if i'm wrong?)

Host Terminal;

docker exec -it immich_server bash
cd upload/
ls -l   <-- can see contents of external library
cd /My_Photos
ls -l   <-- can also see contents of external library
exit
docker exec -it immich_microservices bash
cd upload/
ls -l   <-- can see contents of external library
cd /My_Photos
ls -l   <-- can also see contents of external library
exit

Is this the correct way to check it?

Edit;

I also check the microservices container logs; and notice this after every scan, re-scan, and forced re-scan;

DEBUG [ImmichMicroservices] [LibraryService] Found 0 asset(s) when crawling import paths /My_photos/import/path/one,/My_photos/import/path/two,/My_photos/import/path/twenty

This always comes up as 0 assets, even if they are present?

Is it a permission thing with the docker containers? I don't know what else to check - it worked great for all the initial imports of external library, and i'm not sure when it all went 'offline'.

[matticas]

@alextran1502 any ideas? Anything I can try to investigate?

I'm really trying to avoid making a new library; 2 weeks of scanning, 24 import paths, 175 albums, 210 named people... i really wish to not do all that again.

[matticas]

Just ran another test;

Created a new Import Library using the gui, the folder was nested in my external library called "test_import";

Both server and microservices containers can see it in the 4 locations;

• immich-server: upload/test_import/
  

• immich-server: /My_Photos/test_import/
  

• immich-microservices: upload/test_import/
  

• immich-microservices: /My_Photos/test_import/
  

However, the microservices container "Found 0 assets when crawling import paths" but didn't list the new import path; "test_import".

Is the external library scanning feature having bugs?

[matticas]

Hi etony;

The containers can definitely see the import paths. I've checked this again just now, and as above.

The test_import directory above now works, so scanning is working. But the original large external library isn't been detected.

What permissions should the container see? I just did a sweep through, and i'm getting mixtures of 777, 664, and 700. Where do I set these? (my compose file has the mounts as Read-Only for now).

Thanks!

[etnoy]

Do you see a green check box next to the import path in the web ui?

[matticas]

Yes, on both the original external library, and the new test one, using the Validate button.

[matticas]

Another test just now;

My original external library has 24 import paths.

I just made a new external library, and added the first import part which only has 2 photos. It validated and scanned successfully, and the assets are not offline.

Thus, I suspect the issue is with the database and the original external library? microservices can definitely see the directories. What else can I do to inspect the database? (i've never used a database).

ta

[etnoy]

The core issue must be that the import path is not printed in the logs.

If you enable debug logging, does it print more info? What happens if you remove, save, and readd the path? Assets won't be deleted so don't worry

[matticas]

@etnoy
(sorry replied here, not in thread)

So;

Debug & Verbose logging have been on for some time (i'm watching the microservices container logs). New External Libraries get listed, all the actions get listed, how many assets it finds, faces, etc.

The original External Library that plays up keeps displaying this;

[Nest] 7  - 05/21/2024, 6:46:55 AM     LOG [ImmichMicroservices] [LibraryService] Refreshing library: 16b5bc3b-5d3f-4290-bf56-fdf9589c076a
[Nest] 7  - 05/21/2024, 6:58:34 AM   DEBUG [ImmichMicroservices] [LibraryService] Found 0 asset(s) when crawling import paths /My_Photos/2002 pre-ADFA,/My_Photos/2003 - 2004 ADFA Years,/My_Photos/2005 - 2008 Vic Uni,/My_Photos/2008 - 2010 PSN - Motorbike Times AU,/My_Photos/2010 - 2012 EU,/My_Photos/2012 - Nth America,/My_Photos/2013 AU return,/My_Photos/2014 King Street,/My_Photos/2015-04-28 iPhone 3GS vidoes,/My_Photos/2015 King Street 2,/My_Photos/2016 King Street 3,/My_Photos/2016 Magnolia 10 Bowen Street,/My_Photos/2017 Magnolia 10 Bowen Street,/My_Photos/2018 Magnolia,/My_Photos/2019 Magnolia 10 Bowen St,/My_Photos/2020 Magnolia Isla,/My_Photos/2021 Magnolia Isla big sister - Lewis,/My_Photos/2022 Magnolia Isla Lewis renos continue,/My_Photos/2023 Magnolia Isla Lewis Hilux Camping,/My_Photos/M8 - 28OCT18/,/My_Photos/iphone 19MAR2013/,/My_Photos/Aliboo_Pixel3_Dec19-May23/,/My_Photos/Aliboo_HTCM7_Jul13-May16,/My_Photos/Alison's Photos/
[Nest] 7  - 05/21/2024, 6:58:34 AM     LOG [ImmichMicroservices] [LibraryService] Refreshing library: 16b5bc3b-5d3f-4290-bf56-fdf9589c076a
[Nest] 7  - 05/21/2024, 6:58:34 AM VERBOSE [ImmichMicroservices] [LibraryService] Crawled asset paths paginated
[Nest] 7  - 05/21/2024, 6:58:39 AM VERBOSE [ImmichMicroservices] [LibraryService] Crawled assets have been checked for online/offline status

FYI; my docker mount is (for reference above);
- /media/Shed4/My Photos/:/My_Photos:ro

I just completed the following test;

1. Delete 1 import path from the original External Library.
2. Scanned - nothing.
3. Added it back in - green tick and validated.
4. Scanned - nothing, again zero assets found. Re-added import path can be seen in the import path list above.
5. Force Scanned - nothing, zero assets found. Re-added import path can be seen in the import path list above.

Is it the import paths (and directories themselves) have spaces in them?

I also chmod'd everything in my external library to be 774.

[etnoy]

We did change the glob library a few months ago, the timing sounds about right.

Maybe you can use pgadmin to check the library table in the db.

[matticas]

That was 105.0 - I did read that, this was broken before then. I'm not using globs anyway in the imports, only in the exclusion paths.

[etnoy]

The glob library is used to crawl paths, it's been in the external library code since the start. It's always used, no matter if you glob paths or not

[etnoy]

Gut feeling right now is that we don't correctly handle spaces in import paths. You say it works when doing the test import folder without spaces, right?

[matticas]

Not entirely.
Previous to "asset offline" issues; all import paths had spaces, all imported to the original External Library.
At some point, they all went offline (not sure what version exactly when I upgraded each time).

Test; /My_Photos/test_import/ works.
Test2; /My_Photos/import path one with spaces/ works on a new External Library. Exact same path doesn't on the original External Library.

I've installed a PGadmin container, but i've never used a postgres database, i'm still learning how to connect to the immich postgres docker volume.

[matticas]

Hi etony;

I have pgadmin running and connected to the immich database, however I do not know what I am looking for within it to verify data paths?

Do you have information you could share so I can start searching and trying to validate?

Thanks

[etnoy]

Awesome. You said your had one working library and one that doesn't? Can you check the library table and compare the two?

[matticas]

I will try, I have never used a database or pgadmin, so far I've only connected, I haven't discovered how to display the tables. Edit; Ok, I have found the tables, and can view them. Give me some time to roam about!

[matticas]

Hi @etnoy

So I have discovered;

Table: libraries :: ImportPaths

Any import paths with spaces have " " around them; eg;
{"/docker_container_location/import path with space/","docker_container_location/import path with space2/"}

And import paths without spaces, have no "" around them; eg;
{/docker_container_location/import_test}

Table: assets :: originalPath

values in this column do not have "" around their paths, which the paths include spaces. eg;

'/docker_container_location/import path with space/file.jpg'

However, I have two test libraries, with and without spaces, that work. And the large original External Library that isn't working (with spaces).

Can also confirm, I have assets that are offline, that have no spaces in their import paths or file paths.

What else should I look for?

[matticas]

Hi @alextran1502 @etnoy

From the post above, if I modify the database ImportPath values, and it doesn't fix it, can I revert them back without isssues?

I'm at a loss for what to do to fix (without rebuilding), but I've never worked with databases, and I don't understand enough of the backend of the Immich Microservices to understand the mechanics of the "asset is offline" part.

Any hints or leads to chase to resolve would be much apricated so I can try and find the route cause of the issue.

thanks

[etnoy]

You can modify the import path values in the db, no worries.

@jrasm91, do you have any ideas?

[figadore]

I am having a similar issue. In my case, I did a pg_dumpall from 1.98.2 and then restored to a new container (first to v1.105.1, then to 1.98.2 to reduce the number of variables), and all files have the "This asset is offline." message, even though the server and microservices containers can read those paths. However, there are a handful of iphone videos (where the path has no spaces) that still play, even though the offline error is there and there is no thumbnail. Not all videos play, I can't find anything else to distinguish those that will play and those that won't, and it may just be coincidence/red herring that the ones that do don't have a space in the path. I have tried with and without the pgdata docker volume. I currently have both the working and the broken containers running, what should I look for or test?

Apologies if this is not the same issue.

Edit: I'm running a long force-rescan of all libraries, which is definitely working hard. I'm not sure whether this step is recommended after a restore, but I'll update here if it makes things go back to normal

Edit edit: the force re-scan appears to have fixed it, perhaps this was unrelated. Sorry for the noise.

[matticas]

Yeah my Force Re-Scan doesn't detect any files.

I can even change the database offline boolean, and the Microservices resets it to offline again. Definitely a bug in my opinion.