Sharp - Media Upload Memory Leak

[kasonde]

Bug report

Required System information

• Node.js version: 14.19.1
• NPM version: 6.14.16
• Strapi version: 4.3.8
• Database: PostgreSQL
• Operating system: Linux (Heroku)

Describe the bug

We have a website that depends a lot on media for each product and some products may have a lot of media types like videos audio and images and each one of them may reach 10 to 20 MB. When I try to upload them to Strapi, sometimes due to the size of the media, it not only doesn't upload but also crashes the application itself. We can't upload those files one by one because it affects the user experience a lot so we need to upload them in parallel. This is using the S3 Provider.

Steps to reproduce the behavior

1. Deploy a Strapi app to Heroku
2. Attempt to upload a 10MB file
3. Wait
4. See error in logs

Error logs

2022-09-03T13:28:13.740126+00:00 heroku[web.1]: Process running mem=1280M(249.9%)
2022-09-03T13:28:13.783638+00:00 heroku[web.1]: Error R15 (Memory quota vastly exceeded)
2022-09-03T13:28:13.791665+00:00 heroku[web.1]: Stopping process with SIGKILL

Additional context

If you have noticed in the message they say 1280 megabytes are used and that's for one person, for one upload operation. Let's say there are 10 people uploading at the same time, do we need 10 gigabytes of ram memory?

Internal TID: 1828

[kasonde]

We believe this is related to #12601

[coolista]

thank you, @kasonde Richard.

[Marc-Roig]

@kasonde
I have made a quick investigation about this, I will leave some of my thoughts.

I am am uploading multiple images at the same time (10 of 5MB each) and my heroku dyno Ram gets to 400MB (starting from 200MB). I get a similar behaviour with Railway.

What I am not able to reproduce is that big spike you are reporting. In that project do you use a Dockerfile you could share? Or a Procfile configuration.

The only suspect here is the sharp library. There are many reports of sharp not being "responsible" with memory allocation in Linux machines. They encourage to use jemalloc for memory allocation, but I got spikes of 2GB when trying to install it 😓 Maybe you could try installing the heroku-buildpack-jemalloc and see if there is luck.

[Marc-Roig]

One other thing that could improve the spike of memory is disabling any image optimizations in settings.

[strapi-bot]

This issue has been mentioned on Strapi Community Forum. There might be relevant details there:

https://forum.strapi.io/t/upload-memory-leak-issue/20104/6