Config: Add option to limit resolution of original media files

[adsamcik]

I have a several 500+ megapixel panorama and all of them cause my server to stall during indexing, because it runs out of memory (4+2gb swap). There should be an option to skip processing of large files (they should still be indexed). The option should be configurable based on size in megapixels.

It would be great if it would still generate thumbnails, but unless the algorithm is streamed I don't think that would be a great idea.

Even better would be some worker memory management where it would limit workers if there is a big file to be processed, which it needs more memory for, but I consider option to skip sufficient to solve my issue.

[alexisLefebvre]

Did you set the PHOTOPRISM_ORIGINALS_LIMIT value?

photoprism/internal/config/flags.go

Lines 67 to 72 in 1e023c5

	cli.IntFlag{
	Name: "originals-limit",
	Value: 1000,
	Usage: "file size limit for originals in `MB`",
	EnvVar: "PHOTOPRISM_ORIGINALS_LIMIT",
	},

It's effective to disable generating thumbnail for a video which size is bigger than the limit, I don't know about the photos.

[adsamcik]

I have not, but this would not fundamentally help, because the panorama files might not be that big in size. For example I have 560 mpix panorama which is 35 megabytes. I also have some photos 50+ megabytes that are processed just fine.

[lastzero]

Are those JPEGs? We have raw to JPEG conversion size limits, but not if the original is a JPEG already. What good would indexing an image be if it can't be displayed or analyzed as there are no previews? We can't feed the original to our color detection or TensorFlow. Using a 35 MB file as thumb in search results or albums doesn't make sense either.

You may try reducing the number of workers to 1 so that only one image is indexed at a time.

[adsamcik]

Yes they are JPEGs, well if they are skipped completely that is fine too. I don't mind if they are not indexed, but the problem is that the server runs out of memory and completely dies and has to me manually restarted (even ssh becomes unresponsive). My originals folder contains all the images, so when I run indexing, it matches these too. If I were to move them elsewhere it would just cause fragmentation, because right now I have all the images organised in one place.

What I assumed would happen if there were no thumbs was that there would be some placeholder image and when opened the server would show the original.

For the time being I decided to hide them from prism in an archive. But I think the lockup should be preventable server side (skipping is enough), because every server has a limit.

[lastzero]

Auto detection & configuration can be done in principle, but it takes time to implement and test this in different environments. Not sure when we'll get to it, see roadmap and GitHub issues backlog.

As a quick fix, try reducing the number of workers to 1. In addition, we may be able to add a config option to limit the pixel size of original images.

[alexisLefebvre]

For the time being I decided to hide them from prism in an archive.

You can use the ignore list to ignore some files: https://docs.photoprism.org/user-guide/library/indexing/#ignoring-directories-and-files

[adsamcik]

Auto detection & configuration can be done in principle, but it takes time to implement and test this in different environments. Not sure when we'll get to it, see roadmap and GitHub issues backlog.

As a quick fix, try reducing the number of workers to 1. In addition, we may be able to add a config option to limit the pixel size of original images.

Absolutely, I just wanted to put this on your radar, so that it can be eventually resolved. Limiting the pixel size would definitely solve it. I have only a couple of these large panoramas so limiting workers to 1 would definitely do more harm than good (I currently have 2 which keeps plenty of free ram during imports). I decided to increase swap to 8 gigabytes, which should be more than enough to prevent complete lockups.

You can use the ignore list to ignore some files: https://docs.photoprism.org/user-guide/library/indexing/#ignoring-directories-and-files

This is a pretty good alternative to my archive solution, thanks.