Share photos of a person with other users

[jacksbox]

Which version of recognize are you using?

3.4.1

Enabled Modes

Object recognition, Face recognition

TensorFlow mode

WASM mode

Downstream App

Files App

Which Nextcloud version do you have installed?

27.0.0

Which Operating system do you have installed?

Ubuntu 22.04

Which database are you running Nextcloud on?

postgres

Which Docker container are you using to run Nextcloud? (if applicable)

aio

How much RAM does your server have?

8GiB

What processor Architecture does your CPU have?

arm64

Describe the Bug

Not sure if it's a bug or a feature.
I have two accounts which have access to a lot of shared pictures.
The recognition works quite well and there are just a few Presons which needs to be merged.
But if User A merges two Persons, or moves certain Faces to a Person, this only is applied for this user.
If User B Logs in, the Persons are in the previouse, unmerged state.
Is that intended?

Expected Behavior

Merged Persons and Moved Images should be applied to all users which have access to this images.

To Reproduce

Have 2 users
Share all Pictures
Recognise
Merge some Persons for User A
User B sees the unmerged Persons...

Debug log

No response

[marcelklehr]

Yes this is intended. At some point it may be possible to share "photos of person A" with another user, which is probably what you want, but not at the moment. see #881

[jacksbox]

hm, not exactly, no.
I'll try to explain:

• there are 1000 Images, which are shared between user 1 and 2 (both can access all)
• classification and clustering finished
• clustering resulted in 3 Clusters, of Person A, B, C
• Now User 1 merges Cluster A and B, as this clusters are in fact the same Person
• User 1 has now only 2 Clusters, A and C
• User 2 (with access to the same images) still has Clusters A, B and C

I would expected - better wish ;) - that User 2 now also just has clusters A and C to prevent that user from doing the merge as well (which is easy in this example, but when we talk 10000s of images, there will be a lot of misclassifications which can be corrected)

---

It seems that for now each user uses individual clusters which makes this hard to achieve.
But why not share clusters between all users? Show a user only pictures of a cluster to which the user has access to. If a cluster is empty for a user (because that user does not have access to any picture from it) it can be hidden / not shown.
But I guess that could result in more misclassifications for setups where the users do not have many overlapping persons? Or not? Sorry, just rambling about...

[marcelklehr]

mmmh. Interesting thought. Let me loop in @jancborchardt here to see what he thinks.

[marcelklehr]

I think one thing that would be unexpected would be cluster names. Users should not implicitly share cluster names, even if they have photos of the same person, for privacy reasons. For implicitly sharing the cluster assignments, I don't know, it also feels sort of unexpected that someone else can change my cluster configuration. But then, that's only if a photo is shared already... But what if it's shared read-only?

Also, it might lead to weird problems when two users' photos get assigned to the same cluster that don't depict the same person, but the photos are not shared, then that cluster is kinda tainted without the ability to fix it.

[Forza-tng]

I can see both arguments. However, my personal opinion is to have a choice/setting to share metadata (people, objects, location, etc) with specific groups or people. My use case is family photos which I share with my partner. We have 100k photos from two decades stored and shared on our Nextcloud. It is great if we could jointly work on tagging photos and share the results, as doing duplicate work isn't really feasible.

[rilcy75]

Hi,
Great job!
For me this feature is a must have. As many I use this NC app to share thousands of photo with my (huge) family and friend. Making recognizing and naming people cross NC users is a key features to address my objective.
It could avoid people to use additionnal platform to host dedicated solution for photo sharing.

[Olen]

So, a challenge here is that not only the clusters are per user, but the detections as well, even if the files are shared:

This is an example of an image with 3 faces, shared in a groupfolder with 3 members:

  id   | fileid  |                        path                     |  user_id   | cluster_id                     
-------+---------+-------------------------------------------------+------------+------------
 18465 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |       1163
 18468 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |         -1
 18462 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | XXXX       |        134
 18461 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |        136
 18464 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |        238
 18467 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | YYYY       |         55
 18469 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |         -1
 18463 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |        140
 18466 | 4695116 | __groupfolders/1/2014-12-24_184755-P1000037.JPG | ZZZZ       |          1
(9 rows)

Notice the same file_id and path, but different IDs in the face_detection table.

So each user has an entry in the table for each person (face) in each image. And each of these entries have a unique cluster_id.
The person-identifier (face) is the x/y/vector, so there are some different aproaches possible:

• Remove user_id from the table oc_recognize_face_clusters
• Create a new table oc_recognize_face_clusters_users with two columns: cluster_id and user_id
• By default continue adding one user per cluster as today.
• Allow more user_ids to be added to the same cluster_id in the new table

This will need some additional code to update all the entries in the oc_recognize_face_detections table

1. Create a shared cluster
2. Add users A, B and C to the shared cluster
3. User A moves a "face" (actually combination of file_id, x/y/vector) to shared cluster
4. Magically - the same "face" (same combination of file_id, x/y/vector) is moved to the same shared cluster_id for user B and C

But we need to figure out how to handle different cases.

• What if User B moves the "face" from the shared cluster to a private cluster - or removes the face?
• What if User A moves a non-shared image to the shared cluster?
• What if a shared image file in a shared cluster is moved to a non-shared folder?

Another option is to create what I would call "synced" clusters.

• User A invites User B to sync cluster X
• When the invite is accepted a new cluster Y is created for User B
• All combinations of file_id+vector that exist in cluster X for User A is then duplicated for User B to cluster Y.
• Any further modifications User A does to Cluster X is replicated for User B in Cluster Y
• And any modifications User B does to Cluster Y is replicated to User A in Cluster X

This would need some additional columns in the oc_recognize_face_detections to add a timestamp or version id to ensure that the sync can be done properly, but is a slightly cleaner way as it can basically be done by an external script, and does not need much magic behind the scene.

Non-shared images are no problem here, because if the file_id is not found for User B, the image is just not added to User Bs synced cluster.

But it still have some challenges, like

• What if User B moves a face from the synced cluster to a non-synced cluster. Where should this image be placed for User A?
• There are possibilities for race-conditions where both users try to move a face to a different synced clusters at the same time.

So there is no easy way to do this as far as I can see, and it will require some work to really get it functional.

[marcelklehr]

Please refrain from commenting without adding substantial information. If you wish to express support, use the emoji reactions, please. My team will look into solving issues with a large amount of support every once in a while.

[marcelklehr]

Note, that you do have a little influence over what Nextcloud GmbH works on: We don't promise anything, but every release cycle we try to work on enhancements that get a lot of upvotes, so you may express your support for this by giving this issue an upvote.