Cleanup: Improve logging when deleting related sidecar files

[centralhardware]

---

1. What is not working as documented?

cleanup Removes orphan index entries and thumbnail files - i think is meaning that original folder don't be affected.
but in log i see

delete: removing file <path to my original folder>. file extension is mov

2. How can we reproduce it?

run cleanup command.

3. What behavior do you expect?

no files from original folder will be changed.

4. What could be the cause of your problem?

maybe its's wrong logs. I didn't check I don't have a test instance

5. Can you provide us with example files for testing, error logs, or screenshots?

https://gist.github.com/centralhardware/bf3d264beb99c1abe8bc54c1d287a8a8

6. Which software versions do you use?

last docker version

7. On what kind of device is PhotoPrism installed?

does not matter

8. Do you use a Reverse Proxy, Firewall, VPN, or CDN?

does not matter

[gyto6]

Not a bug to me. I use to run this command, and it doesn't affect my own files. They simply remove their relatives variable in the database tables.

Didi you check if the file's gone and do you now if you deleted it manually yourself?

[centralhardware]

I'm a little afraid to experiment with my files. it's be a little difficult to restore previous state.

[centralhardware]

so i try, and it actually delete file from original folder. this is definitely not what i expect from this command
PS: deleted file MVI_5508.THM

[gyto6]

Does "Deleted" is marked in the "General" Panel ?

[centralhardware]

no

[lastzero]

The log message means the related index entry was deleted, not the original file. You are welcome to suggest a better message, but it's very likely not a bug and as such can't be handled with the highest priority.

[centralhardware]

The log message means the related index entry was deleted,

so why file actually removed?

[lastzero]

That is what I said. We could change it into "delete: removing orphan index file... *

[lastzero]

I got to the bottom of this and found out that cleanup CAN actually delete physical files, especially sidecar files like JPEGs that are automatically created for a RAW, as well as JSON and YAML sidecar files. This was introduced to save disk space and avoid problems as described in the following issue:

• Purge: Can't remove deleted TIFF files #917

The logs are created before anything is actually deleted, so they are not proof that an original was really removed. Note that the cleanup command is intended to remove any remaining artifacts of missing originals, including index entries and sidecar files. So in practice, no original media files should have been deleted, unless the index contains wrong information, e.g. if you put the file there while the cleanup was already running.

Nevertheless, I have refactored large parts of the feature to make sure that no files are deleted in originals when delete is disabled or read-only mode is enabled. Note that some users also use the originals folder for the generated sidecar files. For them, we do not want to break the existing functionality.

[lastzero]

Improved logs when you run photoprism cleanup:

...
DEBU[2022-07-21T18:13:30Z] cleanup: removed thumbnail d/2/d/[...]_500x500_center.jpg 
DEBU[2022-07-21T18:13:30Z] media: debug/drone-av1.webm not found        
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/drone-av1.webm.jpg 
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/drone-av1.yml   
DEBU[2022-07-21T18:13:30Z] media: debug/drone-av1.webm.jpg not found    
DEBU[2022-07-21T18:13:30Z] cleanup: removed debug/drone-av1 from index  
DEBU[2022-07-21T18:13:30Z] media: debug/earth.720p.vp9.webm not found   
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/earth.720p.vp9.webm.jpg 
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/earth.720p.vp9.yml 
DEBU[2022-07-21T18:13:30Z] media: debug/earth.720p.vp9.webm.jpg not found 
DEBU[2022-07-21T18:13:30Z] cleanup: removed debug/earth.720p.vp9 from index 
DEBU[2022-07-21T18:13:30Z] media: debug/DSC_7534.NEF not found          
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/DSC_7534.NEF.jpg 
INFO[2022-07-21T18:13:30Z] media: removed sidecar debug/DSC_7534.yml    
DEBU[2022-07-21T18:13:30Z] media: debug/DSC_7534.NEF.jpg not found      
DEBU[2022-07-21T18:13:30Z] cleanup: removed debug/DSC_7534 from index   
INFO[2022-07-21T18:13:30Z] purge: found no orphan files in index [1.514197ms] 
DEBU[2022-07-21T18:13:30Z] index: updating counts 
DEBU[2022-07-21T18:13:30Z] index: updating covers                       
INFO[2022-07-21T18:13:30Z] removed 3 index entries and 29 thumbnails in 664.565893ms 
...

[lastzero]

These improvements can also be tested with the new preview build that is available shortly! 👍

[centralhardware]

work grate, thanks