WSL2 + Docker causes severe memory leaks in vmmem process, consuming all my machine's physical memory

[davidr-PA]

Version

Microsoft Windows [Version 10.0.19044.1889]

WSL Version

• WSL 2
• WSL 1

Kernel Version

5.10.102.1

Distro Version

Ubuntu-20.04

Other Software

Docker Desktop: 4.11.1

Repro Steps

You simply need to build docker images several times. It results in this:

Note: I don't even have any active Docker containers running. Everything's terminated.

It seems to be the same as this issue, reported 3 years ago and still unresolved: #4166

I never encountered this on MacOS, but it's slowing my development speed down greatly on Windows. It definitely gets worse the more times I build Docker images. It's resulting in build processes slowing down to somewhere between 1/10th and 1/50th the speed they run at when physical memory is available. It makes me need to restart my computer sometimes multiple times per day.

Expected Behavior

Memory gets released after processes terminate.

Actual Behavior

Memory does not get released, resulting in my computer using virtual memory and slowing to a crawl.
Resource usage (with no active containers running - all terminated):

Diagnostic Logs

No response

[labanv]

I have the same issue only by running a few fixed docker images. Within 2 days the memory is 100% used only stopping Docker Destop releases the memory. So I have switched back to Docker Desktop 4.11.0

[kkruit]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

[Drol]

Same here on Windows 11 and Docker Desktop 4.11.1. Running asp.net containers from Visual Studio.

[julestop]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

Same for me. Memory in task manager doesn't add up to total. Limiting the wslconfig file does limit the memory af the task, however, it's like the memory doesn't get returned and after 2-3 days or runtime my computer hits 100% memory usage and I have to reboot.

[LiamKarlMitchell]

Please fix.

[chrisb2244]

Also saw this, unsure if was related to unstarted containers - resource monitor showed ~6 instances of vmmem with 1GB committed and 0KB working set (and one or two with both committed and working, looking more functional).
Task manager showed 99% RAM used, but no really large consumers (just things like Chrome, JDK, Teams, one functioning(ish, given the RAM usage) vmmem, all in 100s MB).

Also running 4.11.1.

@labanv, did rolling back help you?

[Ador-able]

Same here on Windows server 2022 and Docker Desktop 4.11.1. After a few days the server memory takes up 100% the total calculated processes memory in task manager was not equal the total memory being used.

[CurtisVermeeren]

Same issue on Windows 10 Docker Desktop v4.11.1. Left my laptop for a bit and came back to 100% memory usage with 16GB and only one image running.

[davidr-PA]

This is also happening to our team. Seems to have started for us last week. It appears to be maxing out our WSL memory allotted also, as the total calculated processes memory in task manager was not equal the total memory being used.

Same for me. Memory in task manager doesn't add up to total. Limiting the wslconfig file does limit the memory af the task, however, it's like the memory doesn't get returned and after 2-3 days or runtime my computer hits 100% memory usage and I have to reboot.

Nice, I also limited memory allocation in wslconfig. It helped, but doesn't eliminate the issue (it just allows me to get away with building more docker containers, before my machine starts grinding to a halt as it dips into virtual memory).

[JamienAU]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

[labanv]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

for me it works, memory usage is still increasing (some leaks) but sirtainly not as fast as it would hit 100% in 4.11.1 it must be a combination bug of windows vmmem and docker desktop

[julestop]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

Are you back on 4.11.0 or 4.10.0? Looks like you and @labanv are on different versions. I went back to 4.11.0, and seems like my memory still creeps up, maybe more slowly though.

[labanv]

@labanv, did rolling back help you?

Rolled mine back to 4.10.0 and haven't noticed the issue. So far so good for the last ~24 hours.

Are you back on 4.11.0 or 4.10.0? Looks like you and @labanv are on different versions. I went back to 4.11.0, and seems like my memory still creeps up, maybe more slowly though.

I'm back at 4.11.0 but as I say it also increases memory usage but slowly.

[julestop]

I just rolled back from 4.11.0 to 4.10.0 and rebuilt everything, will report back.

[JamienAU]

Have been monitoring memory usage for 48 h now and it's looking good, RAM usage is at the expected level and hasn't creeped up.

[ben186]

Probably related and it will be fixed soon: docker/for-win#12877

[mrgreywater]

Also duplicates #8703

[jonny-circles]

Hi mrgreywater,

I have a fresh install of both WSL and Docker 4.12 and this issue still persists.

[davidr-PA]

@mrgreywater I re-installed with Docker Desktop 4.12, and I still seem to be encountering memory leaks when I build Docker images. Will continue to test as I will definitely be building more Docker images soon - and I will confirm if I see otherwise - but as of now this appears to still be active.

EDIT:
Docker version is:

docker --version
Docker version 20.10.17, build 100c701

[ivsokol]

for me process is based on:

• stopping docker desktop
• running wsl --shutdown
• manually restarting LxssManager service - this frees up all memory
• starting docker desktop
• do it again after couple of hours, when memory is again almost full

[thangnq1001]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

• Check if the file is in UTF-8
• Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run Restart-Service LxssManager
4. Run things again. Inside wsl, run free -mh and check total value, it should be 1GB as configured above

---

Updated: Some folks below (#8725 (comment)) had couple of additional configs in the file that worked for them (kernelCommandLine and autoMemoryReclaim).

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[experimental]
autoMemoryReclaim=gradual

[albcunha]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run Restart-Service LxssManager
4. Run things again. Inside wsl, run free -mh and check total value, it should be 1GB as configured above

Finally!!! This worked great!! Thanks a lot! It was driving me crazy!!!

[riccardogabellone]

Try these steps:

1. in `C:\Users\<yourusername>`, create `.wslconfig` file (skip if already there)

2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run `Restart-Service LxssManager`

4. Run things again. Inside wsl, run `free -mh` and check `total` value, it should be 1GB as configured above

Great man! Thank you!! Works like a charm for me in Windows 11, WSL2 + Docker

One more check I did:

• With no .wslconfig file enter the wsl e see cpus and ram in htop command, actual settings
• After wsl --shutdown, adding that configs and entering the wsl again, htop outputs wanted settings

Anyway, here is official doc

[chabad360]

This should not be the official recommended fix, a memory leak is a memory leak and it needs to be fixed.
I'm not interested in restarting whenever I want to do something more memory hungry.

[LiamKarlMitchell]

Well there are worse issues, when I copy files with Explorer sometimes it locks up all processes/ide etc and I can't even end task them.

[wonson]

Similar issue here.

I have a script with curl dl in a loop, multiple bg process.
I investigated the mem within linux with 'top', and the ram is perfectly fine.

But the ram for Vmmem in Win10 side keeps increasing

[jorgegarcia-ukg]

Same issue here. Running version 4.13.1 (90346).

[ifigueroap]

Same issue here

[themegabyte]

I guess 32Gb ram only cuts about 30 minutes of docker building. I should get 256GB before this is fixed since I tend to develop and test builds for more then 7 hours. Any deals around ya'all?

Edit: I figured my comment was not very helpful.

To add to some context, when I am running VSCODE and normal containers, this is not that much of a problem.

Its very noticeble when I am building containers for my staging builds. It eats way quickly.

Edit 2: I have moved to VMWARE for now. It has much better memory managmenet then WSL2.

[BastLast]

same issue there, I have to restart wsl 2 times a day...

Windows 10 + docker desktop 4.13.1

[RiccardoManzan]

Linux uses free ram as disk cache. It looks like docker (and other apps like intellij) uses a lot this thing. WSL vm has a dynamic ram allocation, nevertheless, the cache is considered as used ram, so, as the free ram is filled with cache, more ram is allocated on the host machine, an it is not released until linux frees the ram. Unfortunately at the moment there is no mechanism to stem this behavior. The only thing you can do is to manually clean the cache using one of the following (from the lightest to the most effective):

sync; echo 1 > /proc/sys/vm/drop_caches
sync; echo 2 > /proc/sys/vm/drop_caches
sync; echo 3 > /proc/sys/vm/drop_caches

Note orevious commands need to be executed by a privileged user

So the best option as for now is to setup a cron that executes one of the previous commands every x minutes

https://devblogs.microsoft.com/commandline/memory-reclaim-in-the-windows-subsystem-for-linux-2/

Yes, that's it. tho you could simply empty the cache every 5 minutes scheduling a crontab task with the following command:

# echo '*/5  * * * * root sync; echo 1 > /proc/sys/vm/drop_caches' >> /etc/crontab

you could also try the values 2 and 3 if you think they could fit your needs.

I'm pretty confused on why this has not been done directly in wsl kernel in order to free unused cache. By the way this looks to me as a decent compromise.

[vietpm]

I got the same error usually. I searched a lot but didn't find the solution for this issue. So I uninstalled Docker Desktop, install Ubuntu from Microsoft store and installed Docker through Ubuntu terminal (using installing guide for Ubuntu, of course). It worked, vmmem process doesn't consume a lot of memory anymore. There's just one tiny note, you need to run command "wsl --shutdown" in cmd or powershell to shut down vmmem completely if you don't need to run Docker anymore.

[atzemis13]

This issue seems to be at least as much a WSL2 issue as a Docker one. Today I was developing in WSL and found that Chrome was reporting out of memory and other programs were crashing. Most interestingly, while vmmem was using 15gb of RAM, task manager showed that I still had ~8gb of free system memory. Not sure what that part was about.

[craigloewen-msft]

Hi folks, we have just put out a new experimental feature in WSL that aims to address this in the latest update. Please update your .wslconfig file to have experimental.autoMemoryReclaim=gradual. This will automatically and gradually release memory back to your Windows machine. Please note that this disables cgroups v1 and could cause errors if you are running the docker deamon in your WSL VM!

More info on this release and the changes can be found here in the blog post. Please tell us what you think!

Related to: #4166

[androiddisk]

Hi folks, we have just put out a new experimental feature in WSL that aims to address this in the latest update. Please update your .wslconfig file to have experimental.autoMemoryReclaim=gradual. This will automatically and gradually release memory back to your Windows machine. Please note that this disables cgroups v1 and could cause errors if you are running the docker deamon in your WSL VM!

More info on this release and the changes can be found here in the blog post. Please tell us what you think!

Related to: #4166

Does it mean it affects the operation of docker?

[nanake]

Does it mean it affects the operation of docker?

If it does. You can run it under cgroup v2.

[vlrevolution]

Hi folks, we have just put out a new experimental feature in WSL that aims to address this in the latest update. Please update your .wslconfig file to have experimental.autoMemoryReclaim=gradual. This will automatically and gradually release memory back to your Windows machine. Please note that this disables cgroups v1 and could cause errors if you are running the docker deamon in your WSL VM!

More info on this release and the changes can be found here in the blog post. Please tell us what you think!

Related to: #4166

How is this useful if you are constantly working? CPU won't be idle then? I am still seeing 26gb vmmem with this setting enabled :(

[mjamilrahman]

Memory Leak, pls Help 😁

[Anguinru]

Hi folks, we have just put out a new experimental feature in WSL that aims to address this in the latest update. Please update your .wslconfig file to have experimental.autoMemoryReclaim=gradual. This will automatically and gradually release memory back to your Windows machine. Please note that this disables cgroups v1 and could cause errors if you are running the docker deamon in your WSL VM!

More info on this release and the changes can be found here in the blog post. Please tell us what you think!

Related to: #4166

This actually worked for me as an stable solution
WSL2: 4.4.0-22621-Microsoft
Docker: 24.0.2, build cb74dfc
Docker Desktop: v4.20.1

[einarpersson]

Does it mean it affects the operation of docker?

If it does. You can run it under cgroup v2.

@nanake Are you saying that the solution to this:

However, this feature does require disabling cgroups v1 in WSL, which can cause some issues. In early testing we noticed that this will break the docker daemon when running it as a service in WSL, and so if you’re using this feature we recommend you use Docker Desktop for your docker needs. We are working with the Docker team to address this in the future.
windows-subsystem-for-linux-september-2023-update

is this?

Added support for running under cgroupv2 on WSL 2. This is activated by adding kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all to your %USERPROFILE%\.wslconfig file in the [wsl2] section. docker-release-notes

[BastLast]

[FonzTech]

Save your files often, otherwise they can become corrupt. Even the Git repo can become corrupt, so always push your commits to remote!!

Most interestingly, while vmmem was using 15gb of RAM, task manager showed that I still had ~8gb of free system memory. Not sure what that part was about.

I also encounter this issue. I have one or two GBs left, but WSL is still crashing while building some Dockerfiles...

Please note that this disables cgroups v1 and could cause errors if you are running the docker deamon in your WSL VM!

Here we go again: fix something, break another.
Basically MS doesn't know how to manage memory for processes...

[TaggartMaher]

Issue persists. Docker consumes all memory available on WSL2.
When building large images, I notice it crashes within 10-20 minutes consistently. I allocated .wslconfig 16gb / 32gb
When switching to 1gb limit, it stops docker from eating everything. But then I can't do anything productive. Currently in the middle of important work, so I just give it 20gb ram so I can work for longer periods. It seems to get worse when I open VSCode with WSL remote connection (only uses 1-2gb for vscode). Considering using dual boot or something. Can't afford to deal with this any longer, been issue for months and gets worse now that I am working with 4-5 containers at a time.

[muellaw]

This issue makes my laptop unusable after a few hours of work. Got 24 GB out of 32 GB RAM allocated to WSL2.
Need to restart WSL2 or reboot to get back to a good state. There sure is room for improvement.

EDIT: adding kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all to %USERPROFILE%\.wslconfig file in the [wsl2] section indeed seems to vastly improve the issue (see the docker release notes 4.17.0. At least in my scenario it seems to help.

TLDR; try to add the following lines to C:\Users\YourUsername\.wslconfig and restart WSL or reboot.

[wsl2]
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[somsomers]

Docker Desktop 4.27.0
Same issue.

[ealcantara22]

Docker Desktop 4.27.2
Same issue

[TheGuardianWolf]

Took me a month to get to this point after diagnosing random server crashes after I leave it alone for a while.

Yep I am seeing this on my media server 😭

[digitalpacman]

Same thing happening to me.

[somsomers]

I'm not sure 100%, but it seems they've fixed it in 4.28.0. Two days after update and no issues

[matej-kaska]

I'm still having that problem. Tried switching to Hyper-V and that didnt work, tried reinstall that didnt work and tried WSL config and still getting 100 % memory usage in two days of running docker containers.

Docker: 4.28.0
WSL: 2.1.5.0

.wslconfig:

[wsl2]
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all
memory=8GB
[experimental]
autoMemoryReclaim=gradual

I don't see that wsl/docker is using the memory in taskmgr.exe or RAMMap.

[Ali-thepro]

I deleted docker recently, but vmmem is still using a lot of memory, couple days ago it was fine now after fresh boot its using 3 gb of memory. I just have 2 wsl terminals open. If I close them it will go down to 1.5.

This is what I get from htop

I have 32 gb ram on my laptop but its getting annoying, its taking too much storage. I tried to limit it in .wslconfig but it causes perfomance issues if I set it too low.

[razeenf]

Still have the same issue on Docker Desktop version 4.28.0

[Ali-thepro]

just ran a small django and front end app on docker 👍, nice stuff wsl team.

[RohitM-IN]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run Restart-Service LxssManager
4. Run things again. Inside wsl, run free -mh and check total value, it should be 1GB as configured above

it worked for me thanks i guess i will need to gist my wslconfig XD

[wsl2]
memory=6GB
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[experimental]
autoMemoryReclaim=gradual

good tested few times using lando for me and stopped started rebuild several times didn't cause the issue

after closing the wsl --shutdown

much much better
Thanks

[Yiidiir]

Try these steps:

1. in C:\Users\<yourusername>, create .wslconfig file (skip if already there)
2. Edit the file:

[wsl2]
memory=1GB # Limits VM memory in WSL 2. If you want less than 1GB, use something like 500MB, not 0.5GB
processors=2 # Makes the WSL 2 VM use two virtual processors

Check if the file is in UTF-8. Some guys had to change the end-of-line type to LF also (Mine was fine with CR-LF).

3. Quit docker and in PowerShell (admin), run Restart-Service LxssManager
4. Run things again. Inside wsl, run free -mh and check total value, it should be 1GB as configured above

it worked for me thanks i guess i will need to gist my wslconfig XD

[wsl2]
memory=6GB
kernelCommandLine = systemd.unified_cgroup_hierarchy=1 cgroup_no_v1=all

[experimental]
autoMemoryReclaim=gradual

good tested few times using lando for me and stopped started rebuild several times didn't cause the issue

after closing the wsl --shutdown much much better Thanks

This seems to work!

[ndalchau]

I uninstalled and stopped using docker desktop ages ago, because of related problems, but was still experiencing this issue with Vmmem hogging memory, CPU, and high power usage. To reclaim my machines resources, I was closing and reopening VS code, which I think did most of the trick.

The .wslconfig additions (autoMemoryReclaim, kernelCommandLine) above have resolved my issue. Thank you!

[craigloewen-msft]

Hi folks, on the latest pre-release WSL versions we've made the 'dropCache' option on by default which should help address this use case. Please upgrade using wsl --update --pre-release and you'll get the autoMemoryReclaim=dropCache behaviour on by default and this should go a long way to fixing it! Thanks.

[prototact]

Hi folks, on the latest pre-release WSL versions we've made the 'dropCache' option on by default which should help address this use case. Please upgrade using wsl --update --pre-release and you'll get the autoMemoryReclaim=dropCache behaviour on by default and this should go a long way to fixing it! Thanks.

I did as was instructed and memory was not reclaimed. I use fedoraremix as my WSL distro and Docker Desktop as a container backend. To test it out, I started a container and installed some large python packages within. After the installation was finished, memory use fell at ~10GB from the max 12GB allowed. The container was otherwise idle, so memory should have been freed.