Race condition with Image Loader makes it unusable in a production environment

[kmdev1]

What version of Next.js are you using?

10.0.9

What version of Node.js are you using?

v12.19.0, v14.16.0

What browser are you using?

Any browser

What operating system are you using?

Any OS

How are you deploying your application?

AWS

Describe the Bug

The default image loader at present seems to suffer from a race condition. Unfortunately this issue has now meant that for us, the pretty awesome functionality, cannot be used at all.

Assuming the image cache does not contain the given image, when making a request the loader will go off and do it's thing and then return the given image and add it to its cache. The problem comes when you have real traffic.

Example scenario:

Lets assume you have a simple page that has 3 images that are utilising next/image. The page has an average traffic of 10 req/s for a given screen size. This page is brand new and the images have never been server before (ie the image's do not yet exist in the cache).

Currently when this above scenario happens, c.30 calls to the original image URL happen. Each one of these images then resolves, gets transformed and then placed into the cache and returned to the user. The important note here is that we now have a cache folder that contains 10 cached files of the same image three times.

When we replicate the above scenario but in more realistic terms (12 images on a new page with average traffic of 25 req/s) this means that we suddenly have a cache folder that contains 25 images x 12 (300 total). This is because when the first request comes in, the code says hey you're not in the cache, lets go get you. The second request comes in and says, hey you're not in the cache (because the first one is still processing), let's go get you and so on and so on.

So why does this cause the container to die? It doesn't; writing and reading from disk is not the only thing that the loader is doing. Http requests, FS, Image manipulation. These things are heavy on not only memory but CPU too.

In the above example we're asking a container to not only go and fetch 300 images per second, we're also manipulating, writing to file storage, serving back. The chances are then by the time these have finished, the next second comes around with another 300 requests, the chances are probably the images still aren't in the cache because the the CPU is rocketed and so has the memory so things just start to back up and get worse and worse until the container falls over completely.

TL;DR:

Requests to non cached images are not aware of each other which means there's a race condition when the same image is requested from the same user within the same time period of it not being in the cache. This then (with enough traffic) spirals into a horrible loop that is non escapable and leads to the containers falling over.

Expected Behavior

I'd expect that when parallel requests are made, that the image loader is directly only called once and is sat behind some sort of batching mechanism (similar to DataLoader) that removes duplicates and is aware of parallel requests and then maps the responses accordingly. This means that the site could require 500 images per second of the same image, but only do all of the hardware intensive stuff once.

To Reproduce

• Create simple page with a single (or multiple) image(s) utilising next/image
• Deploy/run the container so you can see memory/cpu usage
• Start simultaneous request
• Notice Memory/CPU usage climbing
• Notice cache folder starting to drastically grow in size
• If persisted, eventually hardware limitations will be met

[goguda]

I assume this is related to the same thing I'm experiencing with image optimization on anything higher than Next.js v10.0.7. I've tried both v10.0.8 and v10.0.9 but any images that are not in cache seem to cause a memory leak during conversion and memory usage rapidly climbs until my server just runs out of memory and crashes. Conversion also seems much slower overall than previously. Obviously this isn't suitable for production.

I've downgraded back to v10.0.7 for now and I can see memory usage go up by about 10-15MB during a request that would cause several conversions, then the memory gets released like it should. Seems to be a change somewhere after v10.0.7.

[kmdev1]

Hey @goguda,

So I think the issue that you mention could exaggerate the issue that I've raised, however the race condition in itself is a problem within itself unfortunately.

Maybe you'll also notice the same thing as me but when running 10.0.7 locally do you have multiple images in the cache that are the exact same (just with different expiry time stamps - the time stamp is the first bit of the file name)?

To replicate the traffic locally I used a tool called K6 - this replicated 15 users over 10 seconds and I was able to see my cache already being populated with duplicates.

As a side effect of all this over processing this definitely had a hit on hardware for me.

[atom258]

In version 10.1.1, the memory consumption is still too high

[kmdev1]

Hey @shuding,

I have just seen your comment on the other issue about Memory usage. I think the issues that I raise above are related. I believe this still exists in the canary version (unless I'm missing something 😄) I see the more concurrent traffic, the more memory pretty consistently.

Are you able to replicate?

[shuding]

Hi @kmdev1, yes I think these two issues are possibly related, and I'm now trying to reproduce this one. Just want to confirm that you are still able to see the high memory usage in the latest canary with concurrent traffic, right? Asking because in the same situation memory usage will be lower with the fix in #23565.

[PranavSathy]

For what it is worth, our organization is on NextJS 10.0.5 and seeing spikes up to 100% memory utilization and then instance cycling due to OOM on Cloud Run. This started happening once we enabled long-running instances (previously we cycled continuously).

#23188

One other option seems to be moving to the new Canary release with @shuding 's fix:

https://github.com/vercel/next.js/releases/tag/v10.1.3-canary.0

UPDATE: Turns out that after we upgraded to 10.1.0, realized this issue and reverted, we had our package.json as ^10.0.5 , so it still resolved to 10.1.0 due to semver policies, and thus our container image caching utilized the build image with 10.1.0 still.

[kmdev1]

Hey @shuding,

Great piece of work there 😄 - The canary release does indeed fix the overload of memory and it now seems to stay similar to that of 10.0.7 with no real noticeable difference between the two.

Of course I know your work doesn't cater for this ticket so will also drop a line confirming what I've seen to on the original

[kmdev1]

Hey @shuding,

I was just wondering if you'd started any work on this as I probably have a bit of free time this weekend where I could help out if you hadn't started already? 😄

[shuding]

Sorry @kmdev1 that I missed your comment, and yes I'm already working on a PR for this issue 😄

[kmdev1]

No worries at all @shuding! Awesome thank you very much my friend! If there's anything I can do to help just let me know 👍🏻

[balazsorban44]

This issue has been automatically locked due to no recent activity. If you are running into a similar issue, please create a new issue with the steps to reproduce. Thank you.