Thread: Mistakes in face clustering, Shit clusters, Faces not showing up

[IndrekHaav]

I have about 200 photos uploaded to my Nextcloud instance, most of those containing faces. Recognize has processed them all, but only a single person shows up in the People section (of both Photos and Memories apps), with 12 photos.

I don't think this is the same as #588. I have checked the database, and the oc_recognize_face_detections table has 237 records, while oc_recognize_face_clusters only has one. Furthermore, if I manually insert a record into the clusters table and then link a record in the detections table to it, it shows up as a new person. I don't understand why the majority of the detected faces are not added to a cluster.

The same photos, when imported into PhotoPrism (which also uses Tensorflow) resulted in every single detected face showing up.

[marcelklehr]

That would indicate problems with the clustering algorithm. It sounds like it doesn't run at all, though, because it should create at least some clusters, IMO.

[IndrekHaav]

Like I said, it did create one cluster, with 12 faces. Those were all the same person, too, so what work it did, it did correctly.

But just to clarify, the clustering algorithm should be able to create clusters with even a single face? There's no minimum number of faces that need to match before a cluster is created?

Edit: is there any way to run just the clustering job from the command line? With verbosity turned up?

[marcelklehr]

here's no minimum number of faces that need to match before a cluster is created?

There is a minimum. It's currently at 6 detections.

is there any way to run just the clustering job from the command line?

not at the moment, you can manually create an entry in oc_jobs in your database, though, for \OCA\Recgonize\BackgroundJobs\ClusterFacesJob with argument {"userId": "<your user name here>"}

[IndrekHaav]

There is a minimum. It's currently at 6 detections.

I see. That might explain why faces from my photos aren't showing up. Can I ask why this minimum? And is there any way to change it?

[marcelklehr]

@IndrekHaav The clustering algorithm needs a few hyperparameters to work well. From testing it transpired that setting a minimum cluster size improves clustering because it prevents accidental face matches to agglomerate into larger clusters that don't represent a single person (Something I like to call shit clusters). I recommend trying out v3.5.0 first to see if that improves the situation for you, since we're shipping a new clustering algorithm with that release. If that doesn't help and you're adventurous, you can change the min cluster size constant here:

recognize/lib/Service/FaceClusterAnalyzer.php

Line 20 in 015f3c8

public const MIN_CLUSTER_SIZE = 5; // Conservative value: 10

(We've reduced the value from 6 to 5 in v3.5.0 now)

In v3.5.0 there are now convenience occ command for resetting clustering and running clustering manually: occ recognize:reset-face-clusters and occ recognize:cluster-faces

[IndrekHaav]

Thanks for the response!

I tried the new algorithm in 3.5.0. Just to be safe, I wiped all detected faces and clusters from the DB and triggered a full re-crawl.

This time, it created a few more people, but the vast majority of faces were put into a single cluster, seemingly almost randomly. I retried this a few times and also reran the recognize:cluster-faces command a couple of times; the exact numbers varied a bit, but I always ended up with 5-6 clusters, one of them with 80-90 photos of at least a dozen different people (male and female, from infants to 80-year-olds).

In other words, a shit cluster.

Having incorrect faces in the cluster wouldn't be so bad as they can be removed in the UI, but the problem is that there's no way (at least that I could see, in the Photos or Memories apps) to move photos from one person to a new person. One can only move them to an existing person, or remove them from the cluster completely. I tried the latter, but subsequent recognize:cluster-faces commands only put them back (I was hoping they would at least be put into new clusters).

I tried changing MIN_CLUSTER_SIZE, and found that lower values seemed to work better. I set it to 2, and that produced 13 clusters, with overall much better distribution of faces between clusters. That makes sense to me - some people appear in only very few photos, and a higher minimum cluster size might force the algorithm to incorrectly combine them with other faces. I tried setting it to 1, but that produced an error when running the clustering command. I would have liked to see the result, though - even if it errs too far in the other direction, I would rather have more clusters that I have to merge, than fewer clusters that I cannot effectively split.

While I'm messing with the code, is there another constant or parameter that determines how similar the faces have to be to get clustered together?

[marcelklehr]

This time, it created a few more people, but the vast majority of faces were put into a single cluster, seemingly almost randomly.

How many images do you have?

tried the latter, but subsequent recognize:cluster-faces commands only put them back

that would be a bug

I would rather have more clusters that I have to merge

yeah, that makes sense

While I'm messing with the code, is there another constant or parameter that determines how similar the faces have to be to get clustered together?

There is no constant value that governs this. HDBSCAN is an adaptive algorithm that learns from the density patterns of the data. The more files you have, the better the outcome. You could also try playing with MIN_SAMPLE_SIZE.

[IndrekHaav]

How many images do you have?

229. This is just a test set, I'm evaluating Memories + Recognize as a replacement for Photoprism.

I did try setting MIN_SAMPLE_SIZE to 2 as well. That also seemed to improve things a bit - created a few more clusters, and incorrectly grouped faces were at least more reasonable (visually similar, same gender and age group, etc).

[MB-Finski]

How many images do you have?

229. This is just a test set, I'm evaluating Memories + Recognize as a replacement for Photoprism.

I did try setting MIN_SAMPLE_SIZE to 2 as well. That also seemed to improve things a bit - created a few more clusters, and incorrectly grouped faces were at least more reasonable (visually similar, same gender and age group, etc).

Like @marcelklehr already commented, HDBSCAN, as it is implemented, will try to find the most stable clusters from the data regardless of their size. If the data contains a large number of identities (especially multiple identities with fewer face detections than MIN_CLUSTER_SIZE) it'll still try to find the most stable clusters in the data. This can lead to combining multiple identities of similar looking persons. The easiest way to alleviate this issue is to scan in a larger dataset.

Mind you, "similar looking" to the face recognition model may not always be similar looking to you or me. This is especially true in the case of children/infants. The dlib face recognition model hasn't been trained with images of children so they will cause trouble with clustering regardless of the clustering algorithm. (IIRC, Photoprism, for example, had a community effort to retrain their recognition model with datasets containing images of children.)

MIN_SAMPLE_SIZE is basically a probability density (i.e. "face detection density") smoothing factor used by HDBSCAN. Reducing this value too much can lead to statistical noise causing issues with larger datasets. Still, it might be that we'll have to reduce this value going forward; the optimal value in my test dataset may not be optimal for all users. Also, the optimal value will depend, to some degree, on how incremental clustering is implemented as that affects the amount of noise in the data that is being clustered.

@marcelklehr : Besides fine tuning MIN_SAMPLE_SIZE, another way to improve the clustering in this case might be to implement a limit on the maximum size of a cluster. The obvious, but likely(?) not optimal, solution would be to limit the radius of a face cluster. However, limiting the maximum edge length within a cluster (this was implemented in a previous version of the MstClusterer-class but I stripped it since it was not used) may be a better solution since this will specifically limit forming clusters in sparse areas of the face embedding space (since the mutual reachability distance will also be large in these areas). If the latter is implemented, it may help us get away with a larger MIN_SAMPLE_SIZE which is better for users with larger datasets. It may also be that a combination of both of these limits would provide the best user experience.

[IndrekHaav]

@MB-Finski Thanks for the info, that's an interesting read!

However, coming back to the original issue - irrespective of the way the clustering algorithm works, I think there should be a way for the user to review recognised but unclusters faces and, for each one, choose between "not a person" (don't suggest again), "merge with ______" (pick existing cluster) or "new person" (create new cluster).

Or is that something that should be handled by another app like Photos or Memories?

[marcelklehr]

I think this is a good long term plan! The UI will need to be implemented in photos/memories, but recognize will need to provide the backend.

[ced455]

I agree with @IndrekHaav, be able to create a new person is missing when trying to filter out false positive.
I am in the situation where i need to remove face from a person and i cannot find it back in the people tab once done, is this face lost forever ?

[marcelklehr]

remove them from the cluster completely. I tried the latter, but subsequent recognize:cluster-faces commands only put them back (I was hoping they would at least be put into new clusters).

I cannot reproduce this. For me removed face detections are not readded to the same cluster anymore.

[IndrekHaav]

@marcelklehr For me, every time the clustering job ran, the same faces kept getting added to the same person. I ended up deleting the detected faces from the DB, because they were faces I wasn't interested in anyway (random background people, and such).

How would this work anyway? Does the app keep track of clusters that a face has been removed from in some way?

[marcelklehr]

limiting the maximum edge length within a cluster [...] may be a better solution since this will specifically limit forming clusters in sparse areas of the face embedding space

@MB-Finski If you're up for implementing that, I'm happy to merge a pull request (let me know if you need help with git).

[marcelklehr]

How would this work anyway?

When removing a face from a cluster we store the distance from the cluster centroid along with the face and in the future only add it to a cluster if the distance to the cluster centroid is smaller.

[Olen]

Where is this information stored?
Is everything stored in the database, or is there some "AI-magic" stored elsewhere?
I have spent quite a few hours moving photos from one person to another and cleaning up the clusters using "Memories", but obviously only for my own user.
Now I want to replicate the results for other users (shared groupfolders), without repeating all the work.
I can easily run a database-query to change the cluster_id based on file_id, x, y and vector, but if "Move to another person" in the GUI does more than that, I would need to replicate that as well, and that might not be as easy.

[marcelklehr]

The min-distance from cluster centroid is stored in the database for each detection.

[Olen]

Thanks for the quick reply. Does this mean that doing a simple update oc_recognize_face_detections set cluster_id=123 were cluster_id=321 is all it takes?
Or should another process be run as well to calculate and update the new min-distance?
And can/will this process be run automatically/periodically?

[marcelklehr]

Yes, that's all it takes

[marcelklehr]

@MB-Finski I wonder if it would make sense to fall back to DBSCAN clustering for photo collections smaller than x photos, as HDBSCAN results are pretty wild on smaller collections.

[rhatguy]

Coming over to this discussion thread for this since #721 was closed. I'm still experiencing this with V3.7.0. Should we be waiting for the pull in #711 to be released before doing more testing?

[marcelklehr]

Ah, yes, that one only checks for CPU compatibility. Can you post the check at the bottom for node.js?

[rhatguy]

[marcelklehr]

huh. Did you do anything special for this to work, or did it just work out of the box?

[marcelklehr]

Ah, right. The default container is debian, not alpine. Got that wrong.

[rhatguy]

Just confirming...nothing special was done. I've tried to keep my Nextcloud installation as vanilla as possible (other than installing apps inside Nextcloud) to keep it easy to upgrade/maintain.

[MahdiBaghbani]

I have just installed Recognize in Nextcloud and uploaded 100 purely portrait photos, while Recognize tags all them as portraits, people as it should, it fails to find a single face in them.

Face score too low. continuing with next face.
Face is not straight. continuing with next face.

[marcelklehr]

100 photos may not be enough. The clustering algorithm needs a minimum of 100-200 faces to generate useful clusters.

[marlemion]

With recent nextcloud docker container (26), face recognition/clustering must be classified as not working at all. While clusters are created, people are added completely randomly. I wonder whether this feature ever has been tested on a real-life set of pictures? I have about ~30 000 pictures, of which approximately at least 15% have persons on them (I use to be in nature a lot). However, the result is so discouraging that I have not even thought of manually reordering persons and pictures. I remember having set up the dedicated app facerecognition at some point with nextlcoud 24 I think, and that worked way better, but it required a special docker image to be used.

[jmigual]

Indeed, I'm having the same issue. It seems that letting it run longer and feeding it more data works better but not that much better... The clustering algorithm doesn't work that well at all unfortunately...

[marcelklehr]

I wonder whether this feature ever has been tested on a real-life set of pictures?

It has :/

I wonder, are you using WASM mode or normal tensorflow mode?

[marlemion]

I am not using WASM (I am on the official docker image):

[MB-Finski]

Sorry about this taking a while; I've been too busy at work. Anyhow...

So, just a quick recap of what I think is going on here (just in case someone else has time to work on this):

1. Although the underlying HDBSCAN is state-of-the-art (qualitywise) in density based clustering, HDBSCAN is not inherently suitable for incremental processing of embeddings. This means some additional algorithmic complexity (and failure points) when reconciling the results of consecutive clusterings with existing clusters. While the current solution can surely be improved with tweaks in the current code base, ultimately better results could be had by swithing HDBSCAN to FISHDBC. FISHDBC is essentially an approximation of HDBSCAN but with support for incremental processing. FISHDBC would also be computationally much more efficient than the current solution. (Not to mention that efficient large scale clustering would have various applications when working with LLMs).

2. The hyper parameters for the face clustering are currently (likely) far from ideal and tuning them so that everyone gets the best possible results is quite tedious. If you find parameters that work for you, please report back!

3. As discussed above the face recognition deep neural network currently in use (dlib) is not too good with side profile pictures (or children). The density/radius of these "shit clusters" is identical to proper face clusters so from the viewpoint of the clustering algorithm there's no practical way of knowing which is which. At some point we'll probably want to upgrade to a model that has been trained with side profile pictures. Some face recognition model architectures (arc-face) even output a meaningful measure of how sure the model is with the calculated embedding which might be useful.

---

I've made a fork that has (for my personal library) vastly improved performance (mostly improving on points 1 and 2 from above). I'm still running into issues when getting over 30k face detections/user due to the batch prosessing not working correctly in NC 26 but the clustering results are much better for all my users. I intentionally set the hyperparameter such that splitting of identities is more likely than fusing them since this, I think, is the preferred type of error instead of fusing identities.

If you try this fork, please, report back with your results. Only one file needs updating: https://github.com/MB-Finski/recognize/blob/main/lib/Service/FaceClusterAnalyzer.php

[Andreaux]

Indeed, a "Remove from person and don't match anymore" flag would be a nice addition I guess. But then also a way to list "banned" faces so you can reverse that decision later would also be needed :)

[Andreaux]

Maybe simply a "Don't match face" flag and flagging a face would remove it from clusters.

[MB-Finski]

@rhatguy

Thanks for reporting back! It sounds like it is the incremental prosessing that ruins the clustering for you, then. Now that we have a bit more suitable hyper parameters, you should get better results by removing the clusters and running the clustering from cli once you've scanned in all faces (this results in a single HDBSCAN run that clusters all faces at once). Unfortunately, it is inevitable that one or more of these "shit clusters" form even during full HDBSCAN runs (until we switch to some other recognition model). However, these clusters should not consume other identities. If they do, then this suggests that we may need a bit more conservative clustering hyper parameters.

With incremental processing we also have the issue that the noise (clustering rejects) accumulates during successive runs while the number of valid face detections decreases (we only sample previously identified clusters so as to reduce the computational cost). @marcelklehr has an improved version of this fork in the works. This new version also samples the the clustering rejects which should keep the signal to noise -ratio, so to speak, more consistent. This hopefully should reduce the risk of both shit clusters forming and successive runs gobbling up random identities.

EDIT: The flagging of bad clusters/face detections like what you both have suggested is a good idea, IMO.

[Andreaux]

Honestly I feel "shit" clusters are less of a problem. My issue is more with faces straight not getting in any clusters at all and photos with more than one person being attributed to only one of the faces in the picture, only once. I might be wrong but I feel we're missing a lot of faces like that. If a face is in, but in the wrong cluster, you can at least move it, but if it never get in in the first place, that's more of an issue. Honestly, even a cluster with 1 or 2 faces is more useful than leaving it out. To me at least :)

[marcelklehr]

Honestly, even a cluster with 1 or 2 faces is more useful than leaving it out.

The problem is that if we allow a cluster with 1 or 2 faces already it's likely to become a shit cluster.

photos with more than one person being attributed to only one of the faces in the picture

Each photo can only be added to a cluster once, but I cannot reproduce one photo with multiple faces only getting assigned to one cluster.

[Andreaux]

On this topic, I was wondering how Recognize deals with pictures with more than one person in. I might be wrong, but it seems to me as once a picture has been attributed to a person, it is not evaluated for further persons in the image. Could it be? I have a lot of pictures with more people in and once one face is recognized and the picture shows up in that person's "set", the other persons don't seem to show up anywhere, even when there's plenty of pictures with them and their face is clearly visible (even maybe better than the one originally picked). This makes me lose quite a bunch of people.

[marcelklehr]

I can even provide test images

That might be helpful!

I have a lot of photos of my son and daughter in the same picture

For children this might happen because the model embeds children's faces rather close to each other so that both children in the photo will get assigned to the same cluster, which then removes one of them.

[marcelklehr]

which then removes one of them

Come to think about it, we can actually get rid of that now.

[Andreaux]

For children this might happen because the model embeds children's faces rather close to each other so that both children in the photo will get assigned to the same cluster, which then removes one of them.

I get that clustering kids is a challenge (also grouping images from the same kid at different ages, when they have changed a lot in a short time), but I observed the same with adults too.

[Andreaux]

I just did some tests and have to conclude that, in some cases, - you're right - photos with more than 1 recognizable person in are indeed attributed to both persons, but there are definitely cases where this doesn't happen. I'm not sure why though.
I have specific examples (only kids and mixed) where it doesn't happen. I have yet to find examples where all persons are adults and the amount of overlapping images would be high enough to justify a face to be created.
Here's an example:

[marcelklehr]

I think these may be filtered out because the face is rotated too much on one or multiple axes.

[marcelklehr]

I have some news after running a benchmark on the (now improved) clustering algorithm using a part of the IMDb-Face dataset.

Here are the numbers:

Detected faces out of all faces

86% (possibly overestimated)

Clustered faces out of all detected faces

28%

Clustered faces out of detected faces per identity

59%

Identities with clusters out of all identities

87%

Shit clusters out of all clusters

2%

Correctly assigned faces out of all assigned faces per cluster

92%

Note, that the code that these numbers are based on has not been released yet.

[marcelklehr]

Clustered faces out of all detected faces
28%

I'm not sure how we can improve this

[marcelklehr]

The new code is now available as part of v4.1.0 for Nextcloud 26+

[ced455]

Thank you @marcelklehr for all the hard work ! Should users just reset cluster with occ recognize:reset-face-clusters and then run occ recognize:cluster-faces to take advantage of 4.1.0 or shoud we reset everything ?

[marcelklehr]

occ recognize:reset-face-clusters && occ recognize:cluster-faces should be enough. Note that cluster-faces by default will cluster all faces at once, which might be too much for your RAM, so I recommend doing it in batches of e.g. 10000: occ recognize:cluster-faces -b 10000

[rhatguy]

Quick update from 4.1.0. I reset all clusters and re-ran a clustering job across my 41k pictures (~30k faces) in a single clustering run. Without going to in depth the clustering seems MUCH better now. There are still some mistakes, for instance, many pictures of my son and daughter got clustered together and I have one cluster which seems to have picked up random people. I'll have to dig in a bit more to see if there are any recognizable patterns that additional tuning might be able to fix. Nice work though! Recognize seems to be approaching the clustering accuracy that I see from Digikam with this update.

[Andreaux]

I ran into something strange today. I see a different amount of faces in Photos/People and Memories/People. Aside the fact that some faces are blank in Photos and don't open any photos when clicked, I see 1 less persons in Memories. The difference is a person with only 1 photo and that photo is in a hidden folder outside of /Photos. Is there a setting to limit the scope of Recognize to not go out of the Photos (or whatever is set to hold pictures) folder AND not go into hidden folders?

[marcelklehr]

Recognize listens to .nomedia files. Put one in the folder you want it to ignore

[Andreaux]

Not ideal, each Photo application lets you choose a root folder, which is set to /Photos for me. Funnily, if I put a .nomedia file in the other folders outside of Photos, I get errors like "Failed to load some photos" or "No valid root folder found (.nomedia?)" from other apps like Memories (which is instructed to look in /Photos only). I'll leave it like that with the .nomedia files and see on the long term.

[marcelklehr]

You may need to rescan in memories.

[Andreaux]

Probably. Have yet to find how it can be done :)

[iamdoubz]

I have a question, I installed Memories and Recognizer. Memories reports that 85,352 media files have been indexed. I then ran a full classification manually occ recognize:classify. It took approximately 36 hours (for others who are wondering). Now, I am in the Memories app and click on People. My face shows up at least 8 times. My question: how do I combine these into one?

[jmigual]

From memories, you can go inside each face and on the top right menu select, "Merge with other" (or something similar, I'm saying this from memory). You can also move individual photos to another face if they are misclassified by selecting them and going to the top right menu

[iamdoubz]

Thank you for replying! I couldn't find this option in Memories (I think) but did find it within Photos app.

Now better/harder question: does this act retrain the model in Recognizer?

[jmigual]

Kind of but not completely. Each face has a numeric value representing it and the clustering algorithm creates clusters based on the similarity of this numeric value. When you move faces together it takes an "average" of the cluster to decide the value of the face. The more photos you have the more accurate the "average" is to the reality of the face and may sometimes perform better. That is in theory but in practice I've seen very little improvement.

[iamdoubz]

Thank you for the detailed explanation. Theory versus actual is always a fun guessing game. It was just interesting that pictures taken at the same time would have two different clusters for virtually the same pictures.

Last two things then I actually have to start my day job:

• "A minimum of 120 faces per user is necessary for clustering to kick in" It says that I have 31 faces left to process but the background job won't trigger because it needs 120 photos? What command occ can I run to just say "do it anyway"?
• Consequently, when I run an occ recognize:bla command, can I pass a parameter to only use X threads? First time I ran the whole classify item it used all my cores and my server was unusable until it completed and I do not want that to happen again...

[marcelklehr]

What command occ can I run to just say "do it anyway"?

occ recognize:cluster-faces

(You will likely want to use the batch size parameter for this, to avoid blowing up your RAM)

when I run an occ recognize:bla command, can I pass a parameter to only use X threads?

No, there's no cli param for number of cores. You will need to adjust this in the admin settings UI.