[Feature] People splitting (concrete idea)

[Dampfwalze]

I have searched the existing feature requests, both open and closed, to make sure this is not a duplicate request.

• Yes

The feature

Hi,

I know there are already a lot of discussions, touching similar directions of this problem:

• Face detection of similar faces #14596
• Person can be detected twice on one picture #13311
• Manual fixes for facial recognition (identical twins) #3946
• Twins Photos recognized as one person #14137
• BUG: same person more than once in a single photo #12922
• [Feature] Don't tag the same person twice when face recognized twice in one picture #6598
• [Feature] Face Detection - unmerge / split faces and re-evaluate faces #3356

The general consensus is that a single person appearing multiple times in one image is valid behaviour and not a bug (due to mirrors, collages, etc.). The problem arises, when multiple distinct people, such as siblings, collapse into a single identity profile.

Why would this discussion be different? I want to present a concrete idea, how this problem can be solved. In particular, I want to propose a method that can be presented to the user as a button to "Split / Unmerge this person".

Disclaimer: I know this problem is very hard to solve and I don't claim I know the perfect solution. I am just a Computer Science student falling down a rabbit hole and I don't want to claim I know more than the people who built this system. If my terminology is inaccurate, I apologize in advance. Please correct me where I am wrong.

The Problem at hand

Every face runs through the recognition model and produces an embedding (an n-dimensional vector). Immich's task is to find clusters of these embeddings and group them together, each cluster becoming a new person. It does that using the DBSCAN algorithm. Essentially, it uses an euclidean distance with a global distance threshold (Maximum recognition distance). If the closest existing embedding is closer than this threshold, the new embedding is assigned to its cluster. This also means clusters can merge into larger clusters, spanning over a lot of space (think of it as forming a worm). This is common when people change over time (e.g., from infancy to adulthood):

Additionally, Immich allows you to merge multiple clusters manually, introducing gaps, which technically cannot be called one cluster, but still belongs to the same person.

Because infant siblings often share similar facial embeddings, a few ambiguous photos act as a sort of bridge, causing the algorithm to chain both individuals into a single massive cluster:

When embeddings wrongly get assigned to the same cluster, you cannot easily adjust the clustering algorithm to fix the issue: Globally adjusting parameters (like the Maximum recognition distance) destructively affects everything else, fracturing properly clustered people across the entire library. Additionally, we want a permanent solution that also works reliably on new images. A localized solution is needed that focuses on the specific boundary region where the two identities intersect. Note that this region is also localized in the complete cluster and only includes part of the cluster space.

Proposed Solution: Interactive boundary region isolation

I propose to isolate those boundary regions, such that only the embeddings close to the actual problem are included in it. When a user wants to split a person into two people, Immich should try to find the boundary region, where embeddings likely collide. All embeddings falling into it will be processed by a second clustering pass, different from the original pass. This pass is optimized to recognize a specified set of people:

The difficult part is to find this boundary region, since right now, Immich only sees a big cluster and is tasked to separate it into n clusters based on ..., well nothing. That's why a little help from the user is needed. I propose that Immich splits up the big cluster into smaller ones, using the original algorithm, but with a lower distance threshold. That produces a manageable number of sub-clusters and Immich can now ask the user for every sub-cluster, which person it should be assigned to.

In this interaction flow, the user should be presented with multiple examples of a sub-cluster. They should be able to select all people appearing in this cluster. If they select only one person, this cluster is assigned to that person (not globally, temporarily). If they select multiple people, this sub-cluster must be divided again, until every sub-cluster is assigned to exactly one person. After this, Immich knows exactly which embedding belongs to which person.

In the next step we can find an approximate bounding region: We isolate all embeddings that would be grouped together by the original algorithm, but belong to different people. Then we find a bounding region around all of these points and apply a small margin to it. This region will be associated with the clusters in question. Any embedding falling into this region will be evaluated using the second pass. Different clusters will never be connected through this region.

How the second pass could look is up for debate. I am not an expert on the specifics of that.

This solution would leave the existing library intact, preserving existing, manual work and avoiding a lot of additional work.

I would love to hear thoughts from the core team and community on the feasibility of this approach!

Kind regards.

This comment is not written by AI. AI was only used to fact-check it, so I don't tell total nonsense.

Platform

• Server
• Web
• Mobile

[bo0tzz]

It's already possible to manually remove a specific recognized face on an asset, which should cover what you need, no?

That aside, this is a duplicate of #7672.