New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Index: Rescan doesn't remove deleted folders #1010
Comments
Regular rescan doesn't work? |
Unfortunately not, I have tried it a few times, the old photos still there.. |
Also when you manually run photoprism purge in a terminal? |
oh let me see how to do this, I am only using web GUI. |
I am in the docs advanced section, but cannot find CLI guide, could you please help as to where that is? |
Getting Started as well as in our Docker example config file. Look for command reference. |
ah, you just mean simply there is a command |
thank you, that finished as below, and says it deleted a bunch of things, but I can still see them in the gallery.
That said, the machine still has a huge process running
|
Let us know if this is a major bug we need to fix... otherwise we'll tag a new release tomorrow. |
Yes, unfortunately, the photos (at least generated previews of photos) are still available, even though the folder containing them is no longer mounted in the docker at all (after docker restart). I am mounting the originals folder on the host machine (debian10) as read-only, I am not sure if it makes a difference? But I thought a regular rescan would detect removed photos. How can I give you the proper logs etc, to have a look at? |
Do you see any errors in the log? You may also send your logs to hello@photoprism.app @graciousgrey is going to test this again with our current preview build. It's going to be the next stable version (unless we have to fix this issue), so trying it should be safe. |
Might be related to readonly mode, maybe because generated JPEGs don't get removed and stay in the index... |
There was a similar issue recently: Purge: Can't remove deleted TIFF files #917 |
Where are the logs stored. I had a look in the storage mount, and cannot see. In the database maybe? Please let me know, and I will email to you. In the meantime, in the storage docker volume, I can see as attached, in the sidecar folder, the folders with the red box are all completely removed in the originals directory. But still in the sidecar files.. |
sorry, i thought i put a red box around the folders. Basically, from |
Warnings & errors are here: https://demo.photoprism.org/library/errors All logs, incl debug and info messages, are sent to Docker... up to you where / if to store them. |
I have also sent the logs, as a zip file to your email above. Please keep those private, clearly. Thank you again for all of your help, and the great project! |
Maybe also try the cleanup command... |
trying cleanup now, its running |
Still going, will let it run overnight, have a good evening, from Switzerland ;) |
Just added a folder and removed it - works perfectly: Based on this, I've decided to reduced the log level for missing primary files to info as that's what you would expect if you delete them: What database are you using? Maybe there's an issue with the query... didn't have time to look at your full logs yet. |
I keep running cleanup and purge, multiple times successively. They keep removing entires / thumbnails etc each run.. different numbers each time. |
Final update - yes it seems there is a bug: Running this (as below, you get the picture...) finally cleaned everything up. Each one kept purging and cleaning files, decreasing in number each time. Seems it could be a good old binary search issue, that the search just skips items when there are too many.
After 15-20 cycles of the above, above command eventually gave this outcome consistently:
|
Purge already fetches files and photos in batches: https://github.com/photoprism/photoprism/blob/develop/internal/photoprism/purge.go This query might be wrong though (looks like a duplicate WHERE): photoprism/internal/query/photo.go Line 87 in 9e10ba6
|
Started a new preview build so that you may test this if you like - although the issue seems already |
Tested on my own (large) photo library: Found 3 additional photos to delete |
Folders with hidden or missing photos only should be hidden now as well. |
Tested now, same result. But I already had zero results, so I cannot confirm if it is fixed, but indeed things are solved for now for me. Let me "delete" another folder, and try it out. Will come back with more results soon. |
Yes, it did find more, and after one purge and one cleanup they were all gone after one run of each command. Seems this has fixed it! Thank you! |
Excellent, thanks for testing! |
I just had this same issue while running the latest version of PhotoPrism through Docker on an UnRAID server, I'm sorry to say. MariaDB. I had removed 66 folders consisting of 46,000 photos from my physical location. Executing the "purge/cleanup/purge/cleanup/etc..." routine a few times eventually cleared it all out. I'm not sure if there's anything that I can offer in terms of logs, now that it's cleaned up. But, I can definitely suggest that, based on my experience, running purge/cleanup a single time was insufficient under these conditions. |
How long did you wait? Could be a caching issue that eventually resolves itself after indexing & waiting a little. Also possible that related sidecar files were left. In that case the solution is to delete them as well, which is what happens if you use the UI to delete. Might have to implement a delete sidecar command for manually deleted originals. |
I'm a bit untrustsworthy on my memory of the timeline, to be honest, so I'm not sure if it's even of any value to speculate as to how long I waited. It could very well have simply been the issue as you've described it with the caching. Although, I can say that every time I executed the purge/cleanup, it DID find items (for the first few dozen iterations of those commands). So, would that still point towards the cache? Seems like it wouldn't. |
I can offer further information (Sorry -- missed that part in my initial read of your response): The removal of the files occurred outsite of the UI. I physically removed them from the file system instead of "Deleting" them from the UI. |
Rather indicates a problem with orphaned sidecar files... Folders that no longer exist should be hidden / deleted. However, if sidecar entries still exist in the index, this is exactly what can prevent this from functioning. |
Was there an additional command that I should have executed to axe the sidecar orphans? |
We have to focus on completing multi-user features at the moment and work overtime to provide support to as many users as possible. Can't tell you more without digging deeper into details. The code is public, so in theory others could take a look as well... |
|
No worries. I wasn't clear on how to locate the version -- UnRAID simply says "up to date" and I didn't see anything within the UI. Running "photoprism version" from the terminal gave: "211018-e200f322-Linux-x86_64". Looking at the release notes, this looks like October 18, 2021. So, this could be the problem. Still cutting my teeth on the UnRaid experience and even moreso on Docker. Sorry for what was likely a false alarm based on an outdated build. |
In this case, your issue may already be solved |
:) |
Running purge multiple times solved it. Although a bit tangential to the original issue -- would there be a way to help with the documentation where the explanations of purge and cleanup are currently missing? |
I have about 1000 folders (400k photos) in my originals directory, and I deleted (actually moved) about 200 of those to another drive which I don't want to be displayed in Photoprism.
The folders have been removed, but still show up in the gallery, including the ability to view them.
How can I remove these from Photoprism altogether? I realize a complete rescan may work, but the first scan took about two weeks! So I don't want to do that again..
Any advice?
The text was updated successfully, but these errors were encountered: