Count in Photoprism is way lower than source (Apple photos)

[pivot69]

Hello,
I have been trying to figure this one out for weeks now, hoping that I could, and even though I have been able to clear up several issues on the way I have still not reached nirvana on this new migration-adventure...

To sum it all up:
I have 16379 photos/videos in Apple Photos that I have downloaded as originals with metadata. Those I have uploaded to my server and then asked Photoprism to index it all, but Photoprism only counts 16092 photos/videos.
Looking at the (readonly) directory containing the files downloaded from Apple Photos, including live photos and metadata, I count a total of 43911. PhotoPrism counts only 41574.

To clarify:

• When exporting from Apple Photos I did make sure that nothing was left "in the cloud" (for space-saving purposes, an auto-function of iCloud) before initiating the export, and Apple Photos did not report any issues
• The exported photos/videos are all sorted in their "Memories-folders", because after multiple attempts to export the whole "Recents" section of Photos, with all kinds of issues related to this, it was the best option for now
• I have turned off all stacking in Photoprism, which was one of my first major issue with inbuilt functionality, because it was stacking lots of unrelated entries
• All files in the directory and subdirectory are readable by the PhotoPrism docker-user
• There are no reported issues of indexing, at least nothing that I have bee aware of looking at the logs
• And, there is nothing reported in the "Errors" page (located under "Library")
• The server-directory, with its subdirectories, all comes from the same Apple Photos export and is the only one I've initiated to index in Photoprism so that I can try to diagnose the issues Im facing. I'll index the next 80 thousand files when I know its capable
• Photoprism is installed on a clean Ubuntu 22 install, living in its own linux container on a Proxmox VE server
• There are no previous "leftovers" from other attempts to index, I've used all kinds of solution to reset when an attempt failed (reset database doesn't work as good as hoped) and the bullet-proof solution has been to rollback to the zfs snapshot I did after the first docker-compose on the fresh Ubuntu
• The files and directories on the server are read-only for Photoprism, it's correctly mounted and fully available for the lxc and the docker environment, and Photoprism is configured in readonly mode

TRYING TO COUNT:
I've been attempting to count how many files there are in the directory and its subdirectories so that I can compare. I have tried counting and sorting by extensions, and more, to try figuring out the actual photo+video count for the whole directory and its subdirectories. I have not successfully been able to figure out how to count only photos and videos using CLI seeing that there is such a wide range of formats used and they are often not used exclusively for one purpose (heic, mov for live photo, mov for video-recording, png, dng, jeg, and more).

find . -mindepth 1 -type f -name "." -printf x | wc -c
This command gave me the total count of files, and when I tried counting only xmp it gave me a total of 16472 (should be one xmp created per photo and video). Weirdly this is more than what it's supposed to have, indicating duplicates.

I also tried scanning to list the count of each extension present:
Command: find . -type f | sed -e 's/..//' | sed -e 's/.///' | sort | uniq -c | sort -rn
16472 xmp
9659 mov
7448 JPG
5787 HEIC
1444 aae
1121 MOV
772 jpg
437 PNG
359 DNG
204 jpeg
117 mp4
40 heic
36 png
7 dng
5 gif
3 GIF

BOTTOM LINE (KISS):
The reported 16379 photos/videos from Apple Photos still does not correspond with what Photostream reports, even though nothing "seems" to be missing. And I can see no way of figuring it out without going through every single one of them manually.

I see no options in Phtostream to report how many photos/videos there are within a chosen Folder/Originals directory, or any other tool that could make this quality assuring step possible.

So, how can I figure out if hundreds of photos of my kids are missing or included in its final destiny on my server, and also available and shareable in Photostream?

I really (really!) want to use Photoprism as the permanent solution for my gallery of memories*, and stop relying on Apple Photos, and therefore I would like to know for certain that all my photos are actually there before I start pruning and reducing space used on iCloud.

*None of the other options I've mapped and/or tried seems to have the functionality I expect, at least not without major hacking and "maybe"-solutions that would take me days and weeks of frustration to go through

[graciousgrey]

That the count next to search is smaller than the count next to Originals is expected:

Library > Originals shows actual files, whereas Search counts unique photos and videos.

Photos and videos may have more than one file, for example:

A raw file + related jpg file + related xmp file = 3 files, 1 photo
A mp4 file + related jpg file = 2 files, 1 video
It is also possible that multiple .jpg files are stacked because they are related to each other.

If the count next to Originals is smaller than the actual count of files that are physically present your originals folder this might be a hint that the missing ones are duplicates or could not be indexed for some reason. At the moment you need to check the database for duplicates (a Web UI for this is planned).

[pivot69]

Thanks for quick reply!

I've turned off every stacking option before indexing. Photoprism still stack the ones with same image name, if they are in the same directory
So, I would think that the reported count in Apple Photos would be the same as the count next to search?

[lastzero]

See the error logs in Library for file warnings and errors. Duplicates don't have a UI yet, but there's a duplicates database table.

[pivot69]

Nothing in the error-page under Library, unfortunately. Is there a permanent log-file somewhere?
And where is the duplicates database table?

[lastzero]

If you use Docker, all the logs should be there. You can also combine in with a long range of tools specialized in logging. PhotoPrism itself only permanently records errors and warnings. In addition you can query the files table for errors in the file error column and the duplicates table. There is also a long range database frontends available for this so you don't have to use terminal commands.

[pivot69]

Did a "complete rescan" now after activating debug log. When it finished I look at the logs from docker using "docker-compose logs". Grepping for words to see what I can find, then counting it:

docker-compose logs | grep duplicate | wc -l
889

Looking at the actual output, then doing random checks searching for the filenames in Apple Photos. Looks like all, or most of the detected duplicates are indeed duplicates! Congratulation, Photoprism does a better job at detecting duplicates than Apple does!! :-)

[pivot69]

And now for the tedious task of reviewing all 889 filenames to either rename (so that Photoprism will index it) or delete (from Apple Photos, and the local filedump to avoid similar issues and to save diskspace)