Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Thumbs: Skip creating redundant files to save disk space #1874

Closed
CaptnSeraph opened this issue Jan 2, 2022 · 10 comments
Closed

Thumbs: Skip creating redundant files to save disk space #1874

CaptnSeraph opened this issue Jan 2, 2022 · 10 comments
Assignees
Labels
enhancement Refactoring, improvement or maintenance task performance Performance Optimization released Available in the stable release

Comments

@CaptnSeraph
Copy link

CaptnSeraph commented Jan 2, 2022

When setting static size limit to for example maximum, an original with size 1333x2000 results in thumbnails of that size being generated but labelled as 4096x4096_fit and 7680x4320_fit

This is resulting in redundant storage of full size images that should be served as full size images once the original image maximum resolution is reached in thumbnail size.

the thumbnail creation process should really be checking the original image maximum resolution and then only creating thumbnails below that resolution

even if this is to allow for thumbnails to be created of the same size as the original (or the next largest thumbnail size as appropriate) at a higher level of compression, there is no need to create thumbnails that are duplicated but still at the same size, as in my example, the 2560x1600_fit is 25% the disk size of the original image

205kb 2560x1600_fit (actual resolution 1066x1600)
876kb 1333x2000 original
834kb 4096x4096_fit (actual resolution 1333x2000)
834kb 7680x4320_fit (actual resolution 1333x2000)

it makes no sense to keep the two thumbnails in this instance (and probably any other instance where a jpg is being recompressed, it may be different for RAW images, but i doubt that going above original resolution more than once is warranted)

@CaptnSeraph CaptnSeraph added the bug Something isn't working label Jan 2, 2022
@lastzero
Copy link
Member

lastzero commented Jan 2, 2022

Most photographic images will be larger than typical thumbs & screens. You are right in that the current implementation wastes storage with small original JPEGs. It's a known limitation.

We will address it when done with multi-user support as our small team can't work on everything at the same time.

Also, it's possible to upscale small originals in a way that preserves details on larger screens. Just hasn't been implemented yet. There's a GitHub Issue or Discussion for this. Couldn't find it spontaneously, but we're not working today and have friends visiting. Maybe you find and reference it as it then can make sense 👍

@lastzero lastzero added enhancement Refactoring, improvement or maintenance task and removed bug Something isn't working labels Jan 2, 2022
@lastzero lastzero self-assigned this Jan 2, 2022
@lastzero lastzero changed the title If static size limit is above original resolution, multiple redundant copies of thumbnails are created Thumbnails: Don't render thumbs above original resolution to save storage Jan 2, 2022
@CaptnSeraph
Copy link
Author

CaptnSeraph commented Jan 2, 2022

do you mean this?

#1856

If there was a deep zoom toggle i think this issue wouldnt be a problem as i could set a nice low thumbnail size i was content with and then enable "full resolution" via a toggle in the viewer but im trying to understand exactly how the thumbnail dynamic resolution works right now because i dont seem to be able to get good high resolution images out without downloading, im usually limited to vertical resolution

also - go enjoy the new year, theres no rush.

@lastzero
Copy link
Member

lastzero commented Jan 2, 2022

Yes. We're also going to work on the viewer. One step at a time :)

@centralhardware
Copy link

centralhardware commented Jul 5, 2022

thumbnails and original files should not be on the same disks. It is logical to place the thumbnails on a fast disk, while the original files can be located on the hdd. so this optimization may result in slowdowns for some number of users.

@lastzero lastzero added in-progress Somebody is working on this performance Performance Optimization labels Jul 6, 2022
lastzero added a commit that referenced this issue Jul 6, 2022


Signed-off-by: Michael Mayer <michael@photoprism.app>
@lastzero lastzero added please-test Ready for acceptance test and removed in-progress Somebody is working on this labels Jul 6, 2022
@lastzero lastzero changed the title Thumbnails: Don't render thumbs above original resolution to save storage Thumbs: Skip creating redundant files to save disk space Jul 8, 2022
@lastzero
Copy link
Member

lastzero commented Jul 8, 2022

The release notes for our preview build have been updated: https://docs.photoprism.app/release-notes/

Thank you to everyone who helps with testing! We appreciate it very much.

@lastzero
Copy link
Member

lastzero commented Jul 8, 2022

Q: Will redundant thumbnails that have already been created be removed?

A: Not without further work. If this is important to you, then manually delete the cache/humbnails folder and regenerate the thumbs. This will take longer than deleting specific files, but no special logic is required (and most importantly, no required files will be deleted). If you run photoprism cleanup, thumbs without related media files will be deleted, e.g. after they have been modified. The disk space saving could be added there. PRs welcome 🤗

@alexislefebvre
Copy link
Contributor

Could you please provide a list of the files what we could delete? If we know the sizes of the thumbnails, we could delete the redundant thumbnails with something like this: rm cache/thumbnails/**4096x4096_fit*

@lastzero
Copy link
Member

lastzero commented Jul 8, 2022

It depends on the size of the originals, because only the high resolution fit sizes can be skipped. I am AFK now and will be back next week.

@CaptnSeraph
Copy link
Author

If I'm right that the thumbnails are only displayed upto the maximum resolution of the screen they are being displayed on, it may be beneficial for some users who know the maximum resolution of their screens to not exceed 4k to remove the 4096x4096 and the 7680x4320 anyway as I don't believe they would ever be displayed by the current implementation of the viewer.

Unless I'm misunderstanding how the thumbnails work.

@lastzero
Copy link
Member

lastzero commented Jul 9, 2022

The default size limit is 2048, so unless you changed it, you won't have 4k or 8k thumbs anyway.

@graciousgrey graciousgrey added released Available in the stable release and removed please-test Ready for acceptance test labels Jul 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Refactoring, improvement or maintenance task performance Performance Optimization released Available in the stable release
Projects
Status: Release 🌈
Development

No branches or pull requests

5 participants