faces missing / rescanning does not help

[lakemike]

The bug

I am suspecting that my database has accumulated problems since my first docker installation some months ago: While some pictures were initially correctly scanned and faces were shown, these faces have disappeared over time, e.g. after updates and rescans.

A complete redetection/rerecognition with latest version 1.93.3 does not fix the issue. This is actually weird: When I take an affected pictures (including faces), apply a crop to change checksum, I can reupload it and faces will then be recognized. I don't understand why the redetection/rerecognition fails on those other pictures in my database.

The OS that Immich Server is running on

QNAP i386 Linux / Docker

Version of Immich Server

v1.93.3

Version of Immich Mobile App

v1.93.2 build 135

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${EXTPATH_LOCATION}:/mnt/media:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    ports:
      - 2283:3001
    depends_on:
      - redis
      - database
    labels:
      - traefik.enable=true
      - traefik.http.routers.immich.rule=Host(`myserver.domain.tld`)
      - traefik.http.services.immich.loadbalancer.server.port=3001
      - traefik.http.routers.immich.tls.certresolver=leresolver
      - traefik.http.routers.immich.entrypoints=websecure
    networks:
      - immichnet
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - ${EXTPATH_LOCATION}:/mnt/media:ro
      - /etc/localtime:/etc/localtime:ro
    env_file:
      - .env
    depends_on:
      - redis
      - database
    networks:
      - immichnet
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - /share/Container/immich-app/model-cache:/cache
    env_file:
      - .env
    networks:
      - immichnet
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    networks:
      - immichnet
    restart: always

  database:
    container_name: immich_postgres
    image: tensorchord/pgvecto-rs:pg14-v0.1.11
    env_file:
      - .env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - /share/Container/immich-app/pgdata:/var/lib/postgresql/data
    networks:
      - immichnet
    restart: always

networks:
  immichnet:
    driver: bridge

Your .env content

# You can find documentation for all the supported env variables at https://immich.app/docs/install/environment-variables

# The location where your uploaded files are stored
UPLOAD_LOCATION=/share/MyConfig/immich-app/upload

# The location for so called external path
EXTPATH_LOCATION=/share/MyConfig/immich-app/external_path

# The Immich version to use. You can pin this to a specific version like "v1.71.0"
#IMMICH_VERSION=release
IMMICH_VERSION="v1.93.3"

# Connection secrets for postgres and typesense. You should change these to random passwords
TYPESENSE_API_KEY=hidden
DB_PASSWORD=hidden

# The values below this line do not need to be changed
###################################################################################
DB_HOSTNAME=immich_postgres
DB_USERNAME=postgres
DB_DATABASE_NAME=immich

REDIS_HOSTNAME=immich_redis

LOG_LEVEL=log
#LOG_LEVEL=verbose

IMMICH_WEB_URL=http://immich-web:3000
IMMICH_SERVER_URL=http://immich-server:3001

Reproduction steps

1. using docker/ iphone app since 4-5 months
2. both iphone as well as bulk uploads
3. latest version 1.93.3 installed
4. face detection ALL
5. facial recognition ALL

Additional information

No response

[mertalev]

The default for min recognized faces has been increased from 1 to 3, so if the people you notice no longer appear only had 1 or 2 images, then it may not show them. Does this sound like it could be your issue?

[lakemike]

Unfortunately not. I have now set this to 1 and the problem persists.
I should mention another odd thing: Although „min recognized faces“ has been 2, I saw faces where just 1 picture is displayed. So, maybe the missing faces count towards the minimum, but for some reason they are not shown.

[mertalev]

That part is normal. It's a known edge-case that DBSCAN can produce clusters with fewer points than the minimum, but this is benign. It just means there was another person that ended up being a better match.

[mertalev]

It does seem a bit strange that you can upload (almost) the same picture and get a different result. How many users are in this library and how many faces? You can get the face count by running the query SELECT count(*) FROM asset_faces. (Check here if you're not sure how to run this query.)

[lakemike]

Regarding different results: The odd thing is that the database has "forgotten" the results. I.e. when such a given group photo was uploaded around ~October last year, the faces were correctly recognized and shown. Since then, they got lost after updates and rescanning did not bring the back. So, it is somewhat ~assuring that the faces get recognized with a freshly uploaded picture. (Please note I change them slightly to make sure they get a different checksum.) But what happened in between is odd and also the fact that rescanning does not fix it. Is there anyway for the rescanning/rerecognizing process (ALL/ALL) to skip over certain pictures?

Regarding the database statistics:

5 users

29662 images

5656 videos

17278 faces

37 failed file movements

[lakemike]

Hello,
I made some partial progress figuring out what is going on: Archived pictures and pictures in albums do not show up under people. I wonder if this is intended behavior (hope not..)

For personId 660e9c7a-b8c6-4ce7-87e2-10bdfd2c0025 I have uploaded several pictures over time, but only one is shown under people. My impression has been that several pictures are missing.

Indeed, checking this in the database, I can find 5 pictures using the following select statement:

SELECT *
FROM "assets"
WHERE "id" IN (
    SELECT "assetId"
    FROM "asset_faces"
    WHERE "personId" = '660e9c7a-b8c6-4ce7-87e2-10bdfd2c0025'
);

The only difference between the 1 picture which is shown under people for this person and the other 4 pictures which are not shown is that the other 4 pictures are flagged as archived. And to test this, I could confirm that the picture shows up on the people page once isArchived is changed to "f".

=> Is it intended behaviour that archived pictures do not show up with the respective person under people?

Another person 306cb755-34a2-494e-935b-b064afed734f has 18 assets (18 pictures with face recognized), however web interface writes "5 assets" and shows only two pictures.

I could figure out

• one picture can be brought back by changing isArchived to "f".
• all 13 pictures belonging to an album don't show up
• the other 2 pictures not showing up are actually movies

=> Is it intended behaviour that album pictures do not show up with the respective person under people?

For me personally, I use archive to get to a nice timeline. I.e. I would like archived pictures to remain in albums and actually, I would prefer them to also show up under people. I don't find it intuitive that the people page does not show all pictures for a given person. Maybe it could be configured if the people pages (location pages, albums, ..) do show archived pictures or not.

Of note, this is only partial progress.

There are more pictures missing for one of the above persons, which I could find manually in the timeline. It still seems that face detection / recognition results got lost over time and they do not come back with redecting / rerecognizing. Any clues how assets are treated during redetection and rerecognition? Could it be that pictures in albums are being skipped??

[MoriarT3a]

You have done a great job! I was always wondering why there a so less pictures from my kids and wife recognized. With you work: sure all the pictures are in albums to share them with the grandparents.

I hope this will be changed some day. I also prefer to see every photo under the search. Also pictures that were archived and/or in albums.

[lakemike]

I just tested what happens if you archive pictures with GPS coordinates and location tags: They disappear from the page under location tag and also from the world map. This explains why I have several location tags without pictures. 🫠

=> Is it intended behavior that archived pictures no longer show up under their location (neither tag nor world map)?

[mertalev]

Thanks for looking into this!

Is it intended behaviour that archived pictures do not show up with the respective person under people?

Yes, this is intentional. Some people use archival for sensitive content, so we need to be careful about displaying it.

Is it intended behaviour that album pictures do not show up with the respective person under people?

No, definitely not. We don’t filter by this, so I’m not sure how pictures in albums would be excluded. The query for getting assets with a certain person limits to the 1000 newest assets, so I wonder if this is what you’re noticing. Can you reproduce the album issue?

[MoriarT3a]

Is it intended behaviour that archived pictures do not show up with the respective person under people?

Yes, this is intentional. Some people use archival for sensitive content, so we need to be careful about displaying it.
Is the archive searchable? Or is it possible to get a option to chose the behavior?

[lakemike]

Of note, the new web based search returns both unarchived and archived pictures. So, now I can search for a person and see the pictures from the archive. Thank god none of my pictures are sensitive content :)

[mertalev]

I guess we weren't careful enough haha 😅

[mertalev]

Out of curiosity, do you notice any difference for this issue with the current release? My testing with a 400k library showed much better results for recognition recall.

[lakemike]

ok, after moving to a much faster windows PC I could do some more testing.

I think I have figured out what is going on:

• big differences between buffalo_l and antelopev2
• rescanning deletes scan results and the new scan skips over archived pictures
• getting tricked by the fact that archived pictures are not shown under people/ locations

Both effects together explain why I was under the impressions that faces get "forgotten".

I really really wish the users could be allowed to configure how archived pictures get treated.

It's one thing to hide the archived pictures from certain searches.

But current configuration is not intuitive, even confusing:

1. rescanning faces will effectlvely remove faces/people from achived pictures
2. consequently, web search won't find any face/person in the archive! (any search for a person & "archive" give zero hits)
3. mining the archive is not really possible

[mertalev]

It's probably fine for facial recognition to include archived images. I can make that change.

[lakemike]

That would be great. Thanks!

[Chaphasilor]

It's probably fine for facial recognition to include archived images. I can make that change.

Could this be a setting / toggle in the UI instead? Like with search, where we can toggle between showing archived images or not.
For those if us using the archive as a way to filter out crappy images, it really doesn't make sense to include those crappy images in a person overview...

[lakemike]

It's probably fine for facial recognition to include archived images. I can make that change.

Could this be a setting / toggle in the UI instead? Like with search, where we can toggle between showing archived images or not. For those if us using the archive as a way to filter out crappy images, it really doesn't make sense to include those crappy images in a person overview...

The discussion was about the facial recognition, not the default settings for the overview.

If archived pictures are not scanned you will lose their metadata with the next db reset. That means ultimately that you lose the ability to mine your archive.