Limit the number of decoded images in memory

[cozarkd]

Is your feature request related to a problem? Please describe.
I'm trying to optimize 500+ images but it eats all the RAM and then crashes (On WSL2, Windows). On my Mac it just gets stuck and blinking. I've found an issue in the original project with a PR.

No idea if this is hard to implement but would be nice. I can optimize in batches of around 30-50 images but it's not ideal since I need to do those 500 images multiple times.

Thanks for your fork.

[aggregate1166877]

Hi @cozarkd, thanks for the request.

My deadlines are pretty rough at moment, so it might be a few weeks before I can look into this. I feel this issue is nonetheless important so it definitely will get looked at when I get a moment or need a break from other work.

So I can hit the ground running and reproduce the issue quickly, could you tell me the average size and format of the images you're dealing with?

[cozarkd]

My images are 5-15mb and PNG without alpha channel. If I sort by size and get the smaller ones I can take more each time. With the bigger ones I can't process more than 20. 16gb RAM, Windows WSL2 (I think it takes like 70% of total RAM) El mié., 21 feb. 2024 4:46, aggregate1166877 ***@***.***> escribió:

…

Hi @cozarkd <https://github.com/cozarkd>, thanks for the request. My deadlines are pretty rough at moment, so it might be a few weeks before I can look into this. I feel this issue is nonetheless important so it definitely will get looked at when I get a moment or need a break from other work. So I can hit the ground running and reproduce the issue quickly, could you tell me the average size and format of the images you're dealing with? — Reply to this email directly, view it on GitHub <#6 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABRKPG4MP6HB6KTMQDSJJPDYUVUZ5AVCNFSM6AAAAABDSDUHJOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSNJVHAZTKMJXG4> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[aggregate1166877]

Hi @cozarkd - I believe this issue is now fixed. Can you please test with CLI version 0.9.0 0.9.1 when you have a moment?

The application will now only process as many images as you have CPU cores. If you have a crazy amount of cores and run out of memory, I've added a -c option to reduce the amount of concurrent images processed (e.g. to limit to 8 use squoosh-cli -c8 ...).

I also noticed that the terminal output gets badly corrupted with a large amount of files. The CLI will still use the fancy output when processing less than 16 images, but anything more than that will now use a basic / boring output that works correctly with any amount of images.

Testing

I did the following tests using 16 threads, all converting png to webp:

• 1000 images totalling 700MB (maxed out at 7.5GB RAM)
• 300 images totalling 3GB (forgot to measure RAM use)
• 1000 images totalling 10GB (maxed out at 23GB RAM) (maxed out at 8.1GB RAM)

All completed without error.

Edit

There was a memory leak in 0.9.0. Version 0.9.1 no longer has this issue, the application has not exceeded 8.1GB RAM use in any of my tests (though it will probably exceed this amount with images larger than 10MB each at 16+ threads).