[BUG] face recognition is merging my kids' child pictures as if they were all the same people

[pdarcos]

The bug

Hi guys,

First of all thank you for this great project. I'm loving it.

I noticed that the face recognition algo is pretty unreliable when it comes to baby/toddler faces (understandbly so since many babies look alike, specially from the same parents).

In my case the algo has merged together (without me doing anything besides assigning names to my kids' recent pictures) my kids as if they were only one person. I suspect the algo is being misled because they did look alike when they were toddlers.

Is there any way that I can retrain the algo to tell it that they are not the same person? In other words what can we do when the algo makes mistakes?

If there's no easy solution I don't mind digging into the weeds and changing database entries manually or whatever.

And thanks for this great project!

The OS that Immich Server is running on

Ubuntu 22.10

Version of Immich Server

v1.8.1

Version of Immich Mobile App

v1.8.0 build 120

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

version: "3.8"

services:
  immich-server:
    container_name: immich_server
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    command: ["start.sh", "immich"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /media/nas/media/homevideos:/mnt/media/nas/homevideos:ro
      - /media/nas/media/photos:/mnt/media/nas/photos:ro
      
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-microservices:
    container_name: immich_microservices
    image: ghcr.io/immich-app/immich-server:${IMMICH_VERSION:-release}
    # extends:
    #   file: hwaccel.yml
    #   service: hwaccel
    command: ["start.sh", "microservices"]
    volumes:
      - ${UPLOAD_LOCATION}:/usr/src/app/upload
      - /etc/timezone:/etc/timezone:ro
      - /etc/localtime:/etc/localtime:ro
      - /media/nas/media/homevideos:/mnt/media/nas/homevideos:ro
      - /media/nas/media/photos:/mnt/media/nas/photos:ro
    env_file:
      - stack.env
    depends_on:
      - redis
      - database
      - typesense
    restart: always

  immich-machine-learning:
    container_name: immich_machine_learning
    image: ghcr.io/immich-app/immich-machine-learning:${IMMICH_VERSION:-release}
    volumes:
      - model-cache:/cache
    env_file:
      - stack.env
    restart: always

  immich-web:
    container_name: immich_web
    image: ghcr.io/immich-app/immich-web:${IMMICH_VERSION:-release}
    env_file:
      - stack.env
    restart: always

  typesense:
    container_name: immich_typesense
    image: typesense/typesense:0.24.1@sha256:9bcff2b829f12074426ca044b56160ca9d777a0c488303469143dd9f8259d4dd
    environment:
      - TYPESENSE_API_KEY=${TYPESENSE_API_KEY}
      - TYPESENSE_DATA_DIR=/data
      # remove this to get debug messages
      - GLOG_minloglevel=1
    volumes:
      - tsdata:/data
    restart: always

  redis:
    container_name: immich_redis
    image: redis:6.2-alpine@sha256:70a7a5b641117670beae0d80658430853896b5ef269ccf00d1827427e3263fa3
    restart: always

  database:
    container_name: immich_postgres
    image: postgres:14-alpine@sha256:28407a9961e76f2d285dc6991e8e48893503cc3836a4755bbc2d40bcc272a441
    env_file:
      - stack.env
    environment:
      POSTGRES_PASSWORD: ${DB_PASSWORD}
      POSTGRES_USER: ${DB_USERNAME}
      POSTGRES_DB: ${DB_DATABASE_NAME}
    volumes:
      - pgdata:/var/lib/postgresql/data
    restart: always

  immich-proxy:
    container_name: immich_proxy
    image: ghcr.io/immich-app/immich-proxy:${IMMICH_VERSION:-release}
    environment:
      # Make sure these values get passed through from the env file
      - IMMICH_SERVER_URL
      - IMMICH_WEB_URL
    ports:
      - 2283:8080
    depends_on:
      - immich-server
      - immich-web
    restart: always

volumes:
  pgdata:
  model-cache:
  tsdata:

Your .env content

UPLOAD_LOCATION=/media/smb_nas/immich
IMMICH_VERSION=release
TYPESENSE_API_KEY=redacted
DB_PASSWORD=redacted
DB_HOSTNAME=immich_postgres
DB_USERNAME=redacted
DB_DATABASE_NAME=immich
REDIS_HOSTNAME=immich_redis

Reproduction steps

1. As explained above on a fresh install I just tagged a few recent photos of my kids with their names
2. Subsequently searching for their pictures I noticed that immich assigned the wrong person to some of their baby/toddler pictures
3.
...

Additional information

No response

[alextran1502]

Hello, you can try to regenerate the face job again for all, it will start to get the reference data from the most recent time, so the face features of the children are more defined.

[pdarcos]

Thanks, but if I run that job again I think it'll lose all the previously tagged photos and start from scratch no?

[akropp]

It seems that this is not the case. It is definitely merging some of my kids together, and the "reference photo" it picks is a baby photo from several years ago. I initially tagged all the photos as I uploaded them from my phone (in reverse chronological order), and it seemed to do a good job differentiating them. When I re-ran the facial recognition after all of my photos had been uploaded, it seems like it is instead starting from the earliest pictures (or maybe just random order), and so is merging baby photos from different kids together.

[thariq-shanavas]

Optionally, also lower the threshold for faces to be identified as the same person.

[pdarcos]

Thanks. I'm going to try this since I don't want to have to retag all the people again from scratch

[akropp]

Does this take effect for already tagged images? Or do you have to re-run everything (and lose existing tags)?

[thariq-shanavas]

As far as I know, this does not affect already tagged images. You'll have to re-run everything and lose existing tags, unfortunately.

[akropp]

that's what I figured

[pedropombeiro]

This fixed the problem for me.

[go-vegan]

Feature request to unlink faces #5276

[davidpan]

Suggested Features:

• Remove one or more photos from a person's facial recognition
  results.
• Associate one or more photos with a person's facial recognition
  results.

When removing or associating photos, ensure that the corresponding facial recognition results are also cleared, and avoid any re-categorization by existing models.

The function above is also useful for improving future models and learning local algorithm migration.

[davidpan]

Current face recognition algorithms have low recognition rates for Asians and Asian children.
See model description: insightface model zoo

[chefkoch]

Should/Could the age or picture date take also be considered in facial recognition?
Even if the kids look similar, if the date of the picures have a difference of 3 years or so it would propably different kids.

[akropp]

Definitely makes sense.

…

--Adam

On Thu, Mar 21, 2024 at 9:34 PM chefkoch ***@***.***> wrote: Should/Could the age or picture date take also be considered in facial recognition? Even if the kids look similar, if the date of the picures have a difference of 3 years or so it would propably different kids. — Reply to this email directly, view it on GitHub <#4440 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AHIXJVJJYUPDY6BCV57D5GTYZODA3AVCNFSM6AAAAAA54N257GVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DQNZSGQYTK> . You are receiving this because you commented.Message ID: ***@***.***>

[NicholasFlamy]

Changing the threshold as well as the feature to unlink faces #5276 should serve as solutions to this.

[SimpleUserHA]

Hi @bo0tzz , hi @NicholasFlamy ,
#4743 is fine as it is.
Age based face recognition and it's difficulties I understand and would also be a separate topic.

I am not sure if I understood the entities correct but expecting

• a person has a name and a birth date
• a face is identified on a single photo/picture
• a collection of faces, which are similar and matched/assigned to a person

Now the scenario would be:

• a picture of one of my kids where it is one year old could match both kids (collection of faces dated 2021 or 2023) because they were looking similar.
• if the picture is taken in 2023 then it is more likely the second one.

So the date taken might be an additional Factor/ attribute for assigning the to the collection of faces. This might not necessarily mean to detect the age of the face on the picture itself.

Sounds complicated, might be, but just wanted to explain the idea and the thoughts I had about it.

Side note: there is a use case where may idea is working counterproductive: currently faces are assigned correctly when it is a picture (taken years before and hung on the wall) within a picture. (Taken today) :-D

[NicholasFlamy]

So, two main things:

1. It would probably require a new/separate model for age detections.
2. What you mentioned with a picture on the wall. This feature creates new complications on top of it being complicated on its own.

I would just use buffalo_l (or antelopev2 for very diminishing returns with a large performance penalty) and manually fix the edge cases.

[NicholasFlamy]

I would add, I don't think there is an open feature request that asks for age detection to help separate people. If you explain it well and differentiate it from #4743, then I would say it's not a duplicate. #5690 was more focused on what #4743 was focused on so it was mostly a duplicate.

Maybe a title like "Age Detection to Improve Face Recognition" would be good?

I'm not saying that this feature will come any time soon because of a feature request, but because it is separate from simply comparing image date and birth date, I think it should be separate so that people can find it.

[bo0tzz]

I don't think there is an open feature request that asks for age detection to help separate people

There was one and we closed it because it's not feasible.

[NicholasFlamy]

I don't think there is an open feature request that asks for age detection to help separate people

There was one and we closed it because it's not feasible.

I didn't see that. Okay.