Multiple Image Uploads result in memory leak

[iserko]

Describe the bug
Memory leak due to image uploading both with and without responsiveDimensions and sizeOptimization disabled via Media Library settings.

If you upload multiple images at the same, the memory consumption increases according to the number of images you're uploading.

Tested with:

• strapi-provider-upload-local 3.0.1
• strapi-provider-upload-aws-s3 3.0.0

Steps to reproduce the behavior
I run Strapi inside Docker, so I monitor memory usage by running docker stats

Have made a video:

1. Start a fresh Strapi server ... memory usage 93 MB
2. turn off Image optimization and Responsive Dimensions via /admin/settings/media-library 3. go to the Media Library and upload an image 490 kB in size ... memory usage now 122 MB
3. upload another image 149 kB in size ... memory usage now 135 MB
4. enable Image optimization and Responsive Dimension via /admin/settings/media-library
5. upload another image 325 kB in size ... memory usage now 257 MB

Expected behavior
I would expect the additional memory usage to be much lower and stable.

Screenshots
Video: https://drive.google.com/file/d/1wNVkRLKagVQuRp8Vuld6FADrilfmS00n/view?usp=sharing

System

• Node.js version: 12.16.3
• NPM version: 6.14.4
• Strapi version: 3.0.1
• Database: PostgreSQL
• Operating system: Debian Buster running inside Docker

[derrickmehaffy]

@soupette @alexandrebodin ?

On the flip side I will try to see if I can reproduce this outside of docker.

[iserko]

before enhanceFile - 145 MB
after enhanceFile - 154 MB

no change in memory until we get to thumb creation.

after generateThumbnail - 164 MB

after generateResponsiveFormats - 199 MB

if I turn off responsive and optimization ... it just leaves the generateThumbnail bit to leak memory

[iserko]

I wrote a simple test script https://gist.github.com/iserko/ff497e0c9b91b10166dcc1d45d3811a5

# using streams with a minimal memory footprint
root@f93240de99bd:/srv/app# DEBUG=* yarn exec plugins/test.js
yarn exec v1.22.4
  test-sharp at start - 44.6640625 MB +0ms
  test-sharp after initializing the streams - 45.12890625 MB +3ms
  test-sharp after initializing sharp - 45.12890625 MB +2ms
  test-sharp after resizing - 45.12890625 MB +0ms
Image height is 119
Done in 0.24s.

# using buffer adds 6 MB
root@f93240de99bd:/srv/app# DEBUG=* yarn exec plugins/test.js
yarn exec v1.22.4
  test-sharp at start - 44.90234375 MB +0ms
  test-sharp after reading file - 45.6796875 MB +11ms
  test-sharp after resizing - 49.87109375 MB +42ms
  test-sharp after deleting content - 49.87109375 MB +2ms
  test-sharp after deleting newBuffer - 50.19921875 MB +0ms
Done in 0.21s.

[lauriejim]

Thank you @iserko for reporting this issue.
I ping @alexandrebodin about this issue.

The team is quite busy on other topic, can you please try to have a look in the code source to find a potential reason of this issue.
I will appreciate your help on this issue resolution.
Thank you.

[iserko]

@lauriejim as my test from the test script shows, if you use buffers (as Strapi does), you get memory leaks. If you write the code so that it streams all the way through, then there's almost no memory overhead.

[alexandrebodin]

Hi @iserko Would you be interested in making a PR ?

[iserko]

@alexandrebodin can't commit, so it's free for anyone to take it. If I do get time to do it though, can take a stab at it.

[vinod77]

Hi..

I had the same issue: #6869

Looks this memory leak issue is causing due to the Buffer.

https://github.com/strapi/strapi/blob/v3.0.0-beta.20.3/packages/strapi-plugin-upload/services/Upload.js#L82

The default Strapi code uses fs.readFile() which uses Buffer, leads to memory leak.. So I wrote my own code, using fs.createReadStream() which reads/streams the file all the way from the file Storage (S3) itself, in this case there won't be any memory leak issues.

fs.readFile() uses Buffer that holds/leaks the memory to read the files but fs.createReadStream() doesn't.

And i have also removed generateThumbnail: https://github.com/strapi/strapi/blob/v3.0.0-beta.20.3/packages/strapi-plugin-upload/services/Upload.js#L125 which takes huge time to generate thumb.

[markjlaney]

@lauriejim @soupette this is blocking us from upgrading to the new version. This is a high priority bug. Is there any way you can re-prioritize? It causes our server to crash any time multiple users upload videos.

[lynndylanhurley]

Agree with @markjlaney . We just built a site with strapi that cannot be released until this issue is resolved

[alexandrebodin]

Hi, FYI this is in our backlog for this month :)