„Empty“ people / faces detected from Live Photo Video

[raisinbear]

The bug

Hi,
Thanks once more for the fantastic work you guys are doing with Immich!
It took me a while to figure this one out, but is has been there for several versions. Under the people category, empty people keep showing up for me. Empty by means of no assets appearing when clicking on the person. By now, I’m pretty much sure, those are created when analyzing Live Photo Video frame(s) and a face is visible on that that is not visible in the static image itself or is different enough that it isn’t assigned to the same person.
Sorry if that has been brought up elsewhere, but I couldn’t find a matching issue. Also, this may be intentional, but it can be quite hard to assign the correct person from the small preview image, not being able to see the whole image / frame as context.

The OS that Immich Server is running on

Raspbian

Version of Immich Server

v1.101.0

Version of Immich Mobile App

v1.101.0

Platform with the issue

• Server
• Web
• Mobile

Your docker-compose.yml content

IMO not relevant, please let me know if required.

Your .env content

IMO not relevant, please let me know if required.

Reproduction steps

1. Upload Live Photo with person in the (first frame of the?) video part that is not visible in the still image.
2. Observe empty person being created that has no assets linked to it.

Additional information

No response

[aviv926]

Can you run the query?

http://yourimmich.com/api/person/56304fd0-62ba-4b0c-a95e-8512f21ab0b0/assets

When the value 56304fd0-62ba-4b0c-a95e-8512f21ab0b0 is the person's ID.
You can enter the empty face created for him, above the URL will be the ID as in the format I sent above.

[raisinbear]

Sure, this is the reply:
[]

[raisinbear]

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

[aviv926]

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

[raisinbear]

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

I don’t know what should be in the reply for a „valid“ person, but I guess it should be a list of assets, where for this person it is an empty list, correct? So yeah, would make sense that the reply is empty since no assets appear when clicking on the person‘s thumbnail either.

[aviv926]

Sure, this is the reply: []

Just to clarify: This is one of the persons only showing up in a Live Photo accompanying video file as the face is obscured in the still image.

The reply you sent is empty, I guess that's how it appears or is it a miss?

I don’t know what should be in the reply for a „valid“ person, but I guess it should be a list of assets, where for this person it is an empty list, correct? So yeah, would make sense that the reply is empty since no assets appear when clicking on the person‘s thumbnail either.

It should retrieve the person's assets.
The purpose of this was to retrieve the assets and check what the original asset was where the face was identified.

Since it returned an empty result, it is not clear whether it is a bug.

I'll add a label so we can find out what the development team thinks about it.

[raisinbear]

Since it returned an empty result, it is not clear whether it is a bug.

I'll add a label so we can find out what the development team thinks about it.

Got it, thanks. As I said this one face (as many others by now) is only visible in the video part and not the image. In the meantime, I’ll tinker a bit and see what I can find in the db.

[raisinbear]

Ok, so for completeness sake I checked with the person which led to the empty response above. In the db, this person belongs to a face which, as an asset, has the .mov file of a Live Photo, as suspected. I believe, the url response is empty, as Live Photo videos are not assets to be supplied (shown, listed) independently from the still image, by the framework, but that is pure guesswork.

[aviv926]

Ok, so for completeness sake I checked with the person which led to the empty response above. In the db, this person belongs to a face which, as an asset, has the .mov file of a Live Photo, as suspected. I believe, the url response is empty, as Live Photo videos are not assets to be supplied (shown, listed) independently from the still image, by the framework, but that is pure guesswork.

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked?
If you don't want to reveal a face you can take a picture of an ad face or something.

[raisinbear]

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Thanks for the info! I’m afraid I can’t share any of the original photos, there are too many people plus infants in them to ask for permission. I’ll try and capture one from an ad or something as you suggested. But getting the scenario right may be challenging. If I can’t produce one, I’ll just test with the next release and report back.

[raisinbear]

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Ok, took me a while to get it right, but I got one that is working. However, I don't know how else to share it but as the .heic + .mov files. Not sure how you would combine them back into a live photo and if the effect persists then. Please let me know if there is another way to properly get this to you.

IMG_4304.zip

[raisinbear]

And maybe one other thing. I stumbled upon another issue while trying to produce a sample live photo, which might make it hard / impossible for anyone else to reproduce the issue: Upon uploading a single live photo, I would regularly get this kind of error signature.

Error

immich_microservices     | [Nest] 7  - 04/13/2024, 10:03:58 AM   ERROR [JobService] Unable to run job handler (thumbnailGene
ration/generate-preview): Error: ffprobe exited with code 1
immich_microservices     | ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
immich_microservices     |   built with gcc 12 (Debian 12.2.0-14)
immich_microservices     |   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --d
isable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enab
le-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --en
able-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libm
p3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --ena
ble-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=a
rm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga
immich_microservices     |   libavutil      58.  2.100 / 58.  2.100
immich_microservices     |   libavcodec     60.  3.100 / 60.  3.100
immich_microservices     |   libavformat    60.  3.100 / 60.  3.100
immich_microservices     |   libavdevice    60.  1.100 / 60.  1.100
immich_microservices     |   libavfilter     9.  3.100 /  9.  3.100
immich_microservices     |   libswscale      7.  1.100 /  7.  1.100
immich_microservices     |   libswresample   4. 10.100 /  4. 10.100
immich_microservices     |   libpostproc    57.  1.100 / 57.  1.100
immich_microservices     | upload/library/ea8f7f01-33a1-4b8a-85d4-15ab6689bc27/2024/04/IMG_4309.MOV.mov: No such file or dir
ectory
immich_microservices     |
immich_microservices     | [Nest] 7  - 04/13/2024, 10:03:58 AM   ERROR [JobService] Error: ffprobe exited with code 1
immich_microservices     | ffprobe version 6.0.1-Jellyfin Copyright (c) 2007-2023 the FFmpeg developers
immich_microservices     |   built with gcc 12 (Debian 12.2.0-14)
immich_microservices     |   configuration: --prefix=/usr/lib/jellyfin-ffmpeg --target-os=linux --extra-version=Jellyfin --disable-doc --disable-ffplay --disable-ptx-compression --disable-static --disable-libxcb --disable-sdl2 --disable-xlib --enable-lto --enable-gpl --enable-version3 --enable-shared --enable-gmp --enable-gnutls --enable-chromaprint --enable-opencl --enable-libdrm --enable-libass --enable-libfreetype --enable-libfribidi --enable-libfontconfig --enable-libbluray --enable-libmp3lame --enable-libopus --enable-libtheora --enable-libvorbis --enable-libopenmpt --enable-libdav1d --enable-libsvtav1 --enable-libwebp --enable-libvpx --enable-libx264 --enable-libx265 --enable-libzvbi --enable-libzimg --enable-libfdk-aac --arch=arm64 --cross-prefix=/usr/bin/aarch64-linux-gnu- --toolchain=hardened --enable-cross-compile --enable-rkmpp --enable-rkrga
immich_microservices     |   libavutil      58.  2.100 / 58.  2.100
immich_microservices     |   libavcodec     60.  3.100 / 60.  3.100
immich_microservices     |   libavformat    60.  3.100 / 60.  3.100
immich_microservices     |   libavdevice    60.  1.100 / 60.  1.100
immich_microservices     |   libavfilter     9.  3.100 /  9.  3.100
immich_microservices     |   libswscale      7.  1.100 /  7.  1.100
immich_microservices     |   libswresample   4. 10.100 /  4. 10.100
immich_microservices     |   libpostproc    57.  1.100 / 57.  1.100
immich_microservices     | upload/library/ea8f7f01-33a1-4b8a-85d4-15ab6689bc27/2024/04/IMG_4309.MOV.mov: No such file or directory
immich_microservices     |
immich_microservices     |     at ChildProcess.<anonymous> (/usr/src/app/node_modules/fluent-ffmpeg/lib/ffprobe.js:233:22)
immich_microservices     |     at ChildProcess.emit (node:events:518:28)
immich_microservices     |     at ChildProcess._handle.onexit (node:internal/child_process:294:12)
immich_microservices     | [Nest] 7  - 04/13/2024, 10:03:58 AM   ERROR [JobService] Object:
immich_microservices     | {
immich_microservices     |   "id": "5e86106d-a115-4a82-b946-ff3cd2c10cc2",
immich_microservices     |   "source": "upload"
immich_microservices     | }
immich_microservices     |
immich_microservices     | [Nest] 7  - 04/13/2024, 10:04:02 AM     LOG [MediaService] Successfully generated JPEG image thumbnail for asset ffb84916-803d-4e25-8ee3-ee8169a479fd
immich_microservices     | [Nest] 7  - 04/13/2024, 10:04:02 AM     LOG [MediaService] Started encoding video 5e86106d-a115-4a82-b946-ff3cd2c10cc2 {"inputOptions":[],"outputOptions":["-c:v h264","-c:a aac","-movflags faststart","-fps_mode passthrough","-map 0:0","-map 0:1","-v verbose","-vf scale=720:-2,format=yuv420p","-preset veryfast","-crf 28"],"twoPass":false}
immich_machine_learning  | [04/13/24 08:04:02] INFO     Setting 'buffalo_l' execution providers to
immich_machine_learning  |                              ['CPUExecutionProvider'], in descending order of
immich_machine_learning  |                              preference
immich_machine_learning  | [04/13/24 08:04:02] INFO     Loading facial recognition model 'buffalo_l' to
immich_machine_learning  |                              memory
immich_microservices     | [Nest] 7  - 04/13/2024, 10:04:09 AM     LOG [MediaService] Successfully generated WEBP image thumbnail for asset ffb84916-803d-4e25-8ee3-ee8169a479fd
immich_microservices     | [Nest] 7  - 04/13/2024, 10:04:14 AM     LOG [MediaService] Successfully encoded 5e86106d-a115-4a82-b946-ff3cd2c10cc2

While the thumbnail and video would show up in the ui eventually, facial recognition was not performed, at least for the video part of the assets that produced this error. What I did was capture a couple of other images that were eventually uploaded first and produced the above error. The Live Photo in question was then processed correctly, meaning the empty face was detected.
I assume this error would be a separate issue. Given that it appears sometimes, but not always and only for the first image or so when uploading a couple of images gives me the impression that there is a race condition somewhere..

[aviv926]

There is a possibility that it was fixed with the merge of #8724, could you upload a sample file so that it can be checked? If you don't want to reveal a face you can take a picture of an ad face or something.

Ok, took me a while to get it right, but I got one that is working. However, I don't know how else to share it but as the .heic + .mov files. Not sure how you would combine them back into a live photo and if the effect persists then. Please let me know if there is another way to properly get this to you.

IMG_4304.zip

I was unable to restore the file back to a live image
Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

[raisinbear]

I was unable to restore the file back to a live image Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

Thanks a lot for trying! I’ll just wait for the new release then and see if it was fixed.

[aviv926]

I was unable to restore the file back to a live image Neither with ffempg nor with https://github.com/RhetTbull/makelive (it requires mac os)

Thanks a lot for trying! I’ll just wait for the new release then and see if it was fixed.

No problem, let me know if it worked for you so we know whether to close this issue