Index: Rescan doesn't remove deleted folders

[maltokyo]

I have about 1000 folders (400k photos) in my originals directory, and I deleted (actually moved) about 200 of those to another drive which I don't want to be displayed in Photoprism.

The folders have been removed, but still show up in the gallery, including the ability to view them.

How can I remove these from Photoprism altogether? I realize a complete rescan may work, but the first scan took about two weeks! So I don't want to do that again..

Any advice?

[lastzero]

Regular rescan doesn't work?

[maltokyo]

Unfortunately not, I have tried it a few times, the old photos still there..

[lastzero]

Also when you manually run photoprism purge in a terminal?

[maltokyo]

oh let me see how to do this, I am only using web GUI.

[maltokyo]

I am in the docs advanced section, but cannot find CLI guide, could you please help as to where that is?
https://docs.photoprism.org/user-guide/settings/advanced/

[lastzero]

Getting Started as well as in our Docker example config file. Look for command reference.

[maltokyo]

ah, you just mean simply there is a command photoprism purge - sorry, I will try this.
Thank you for your help.

[maltokyo]

thank you, that finished as below, and says it deleted a bunch of things, but I can still see them in the gallery.

WARN[2021-02-06T19:55:07Z] photo: photo pqnd7992yynrln6m has no jpegs (set new primary) 
INFO[2021-02-06T19:55:07Z] purge: searching index for hidden media files 
INFO[2021-02-06T19:55:20Z] purge: removed 22913 files and 33500 photos in 8m12.047469537s 
INFO[2021-02-06T19:55:20Z] closed database connection  

That said, the machine still has a huge process running darktable-cli as below, so maybe it still needs time. Ill leave it running overnight and see.

top - 21:22:28 up  2:48,  0 users,  load average: 18.02, 13.15, 7.19
Tasks:   5 total,   2 running,   3 sleeping,   0 stopped,   0 zombie
%Cpu(s): 57.8 us, 18.5 sy,  0.0 ni, 15.5 id,  8.1 wa,  0.0 hi,  0.1 si,  0.0 st
MiB Mem :   7925.8 total,   5259.1 free,   2371.6 used,    295.1 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used.   5293.1 avail Mem 

  PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                         
 4573 root      20   0  144016  53856  33484 R  12.6   0.7   0:00.38 darktable-cli 

[lastzero]

Let us know if this is a major bug we need to fix... otherwise we'll tag a new release tomorrow.

[maltokyo]

Yes, unfortunately, the photos (at least generated previews of photos) are still available, even though the folder containing them is no longer mounted in the docker at all (after docker restart).

I am mounting the originals folder on the host machine (debian10) as read-only, I am not sure if it makes a difference? But I thought a regular rescan would detect removed photos.

How can I give you the proper logs etc, to have a look at?

[lastzero]

Do you see any errors in the log? You may also send your logs to hello@photoprism.app

@graciousgrey is going to test this again with our current preview build. It's going to be the next stable version (unless we have to fix this issue), so trying it should be safe.

[lastzero]

Might be related to readonly mode, maybe because generated JPEGs don't get removed and stay in the index...

[lastzero]

There was a similar issue recently: Purge: Can't remove deleted TIFF files #917

[maltokyo]

Where are the logs stored. I had a look in the storage mount, and cannot see. In the database maybe? Please let me know, and I will email to you.

In the meantime, in the storage docker volume, I can see as attached, in the sidecar folder, the folders with the red box are all completely removed in the originals directory. But still in the sidecar files..

[maltokyo]

sorry, i thought i put a red box around the folders. Basically, from _queenie folder down to and including the _WIP_q folder, these are all not existing in the originals folder.

[lastzero]

Warnings & errors are here: https://demo.photoprism.org/library/errors

All logs, incl debug and info messages, are sent to Docker... up to you where / if to store them.

[maltokyo]

There are thousands of this error in library/errors

[maltokyo]

I have also sent the logs, as a zip file to your email above. Please keep those private, clearly. Thank you again for all of your help, and the great project!

[lastzero]

Maybe also try the cleanup command...

[maltokyo]

trying cleanup now, its running

[maltokyo]

Still going, will let it run overnight, have a good evening, from Switzerland ;)

[lastzero]

Just added a folder and removed it - works perfectly:

Based on this, I've decided to reduced the log level for missing primary files to info as that's what you would expect if you delete them:

What database are you using? Maybe there's an issue with the query... didn't have time to look at your full logs yet.

[maltokyo]

Final update - yes it seems there is a bug:

Running this (as below, you get the picture...) finally cleaned everything up. Each one kept purging and cleaning files, decreasing in number each time. Seems it could be a good old binary search issue, that the search just skips items when there are too many.

photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup && photoprism purge && photoprism cleanup

After 15-20 cycles of the above, above command eventually gave this outcome consistently:

INFO[2021-02-08T12:06:54Z] purge: searching index for hidden media files 
INFO[2021-02-08T12:07:02Z] purge: removed 0 files and 0 photos in 1m5.9222824s 
INFO[2021-02-08T12:07:02Z] closed database connection                   
INFO[2021-02-08T12:07:05Z] cleanup: read-only mode enabled              
INFO[2021-02-08T12:07:10Z] cleanup: removed 0 index entries and 0 orphan thumbnails in 6.149076047s 
INFO[2021-02-08T12:07:10Z] closed database connection  

[lastzero]

Purge already fetches files and photos in batches: https://github.com/photoprism/photoprism/blob/develop/internal/photoprism/purge.go

This query might be wrong though (looks like a duplicate WHERE):

photoprism/internal/query/photo.go

Line 87 in 9e10ba6

func ResetPhotoQuality() error {

[lastzero]

Started a new preview build so that you may test this if you like - although the issue seems already fixed solved?

https://docs.photoprism.app/getting-started/updates/#development-preview

[lastzero]

Tested on my own (large) photo library: Found 3 additional photos to delete 👍

[lastzero]

Folders with hidden or missing photos only should be hidden now as well.

[maltokyo]

Tested now, same result. But I already had zero results, so I cannot confirm if it is fixed, but indeed things are solved for now for me. Let me "delete" another folder, and try it out. Will come back with more results soon.

[maltokyo]

Yes, it did find more, and after one purge and one cleanup they were all gone after one run of each command. Seems this has fixed it! Thank you!

[lastzero]

Excellent, thanks for testing!

[kevinmkr]

I just had this same issue while running the latest version of PhotoPrism through Docker on an UnRAID server, I'm sorry to say.

MariaDB.

I had removed 66 folders consisting of 46,000 photos from my physical location.

Executing the "purge/cleanup/purge/cleanup/etc..." routine a few times eventually cleared it all out.

I'm not sure if there's anything that I can offer in terms of logs, now that it's cleaned up. But, I can definitely suggest that, based on my experience, running purge/cleanup a single time was insufficient under these conditions.

[lastzero]

How long did you wait? Could be a caching issue that eventually resolves itself after indexing & waiting a little.

Also possible that related sidecar files were left. In that case the solution is to delete them as well, which is what happens if you use the UI to delete. Might have to implement a delete sidecar command for manually deleted originals.

[kevinmkr]

I'm a bit untrustsworthy on my memory of the timeline, to be honest, so I'm not sure if it's even of any value to speculate as to how long I waited. It could very well have simply been the issue as you've described it with the caching.

Although, I can say that every time I executed the purge/cleanup, it DID find items (for the first few dozen iterations of those commands). So, would that still point towards the cache? Seems like it wouldn't.

[kevinmkr]

I can offer further information (Sorry -- missed that part in my initial read of your response):

The removal of the files occurred outsite of the UI. I physically removed them from the file system instead of "Deleting" them from the UI.
The purge/cleanup cycles occurred in the terminal.
I DID run the re-index command from the UI a couple of times, including with the "Complete Rescan" checked. But, again, the order of when I did what and the time in between the operations has been "purged" from my brains, unfortunately.

[lastzero]

Rather indicates a problem with orphaned sidecar files... Folders that no longer exist should be hidden / deleted. However, if sidecar entries still exist in the index, this is exactly what can prevent this from functioning.

[kevinmkr]

Was there an additional command that I should have executed to axe the sidecar orphans?

[lastzero]

We have to focus on completing multi-user features at the moment and work overtime to provide support to as many users as possible. Can't tell you more without digging deeper into details.

The code is public, so in theory others could take a look as well... 😉

[lastzero]

• Needless to say, we tested it all before releasing it, so it should generally work. It could be an edge case, 46,000 photos is a lot and no folder structure is the same.
• Note that there was a bug in a previous version that prevented certain index entries from being permanently removed when you delete a photo.
• So the behaviour depends not only on the version you are currently using, but also on when you started deleting photos.

[kevinmkr]

No worries. I wasn't clear on how to locate the version -- UnRAID simply says "up to date" and I didn't see anything within the UI. Running "photoprism version" from the terminal gave: "211018-e200f322-Linux-x86_64". Looking at the release notes, this looks like October 18, 2021. So, this could be the problem.

Still cutting my teeth on the UnRaid experience and even moreso on Docker.

Sorry for what was likely a false alarm based on an outdated build.

[lastzero]

In this case, your issue may already be solved 💎

[kevinmkr]

:)

[ChenSun-Phys]

Running purge multiple times solved it. Although a bit tangential to the original issue -- would there be a way to help with the documentation where the explanations of purge and cleanup are currently missing?