Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

WSL 2 consumes massive amounts of RAM and doesn't return it #4166

Open
LordMonoxide opened this issue Jun 17, 2019 · 93 comments
Assignees

Comments

@LordMonoxide
Copy link

@LordMonoxide LordMonoxide commented Jun 17, 2019

  • Your Windows build number: 18917

  • What's wrong / what should be happening instead: WSL 2 starts using huge amounts of RAM after a while, just using it like normal. At the moment I'm using phpstorm, and did a dump/load of a database. Vmmem is using 7 GB of my 16 GB of RAM and not returning any, even though Ubuntu is actually using much less. I have seen it grow until nearly 100% of my system memory is in use, and it will not release it until I shut down the WSL 2 VM.

This may or may not be related to #4159

corey@Corey-Laptop:/mnt/c/WINDOWS/system32$ vmstat -s
     15235516 K total memory
       920348 K used memory
      1886048 K active memory
      6434312 K inactive memory
      6606548 K free memory
        76280 K buffer memory
      7632340 K swap cache
            0 K total swap
            0 K used swap
            0 K free swap
       163729 non-nice user cpu ticks
          298 nice user cpu ticks
        13177 system cpu ticks
     68988300 idle cpu ticks
         8962 IO-wait cpu ticks
            0 IRQ cpu ticks
        10022 softirq cpu ticks
            0 stolen cpu ticks
      1481417 pages paged in
      6792976 pages paged out
            0 pages swapped in
            0 pages swapped out
      1079177 interrupts
      5131981 CPU context switches
   1560599814 boot time
         8772 forks
@Drakota

This comment has been minimized.

Copy link

@Drakota Drakota commented Jun 17, 2019

Same here with Docker running on Ubuntu 18.04
image

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Jun 17, 2019

Thanks for opening the issue. We have a fix for this in the works.

@mithunshanbhag

This comment has been minimized.

Copy link

@mithunshanbhag mithunshanbhag commented Jun 19, 2019

After moving to WSL2, VmMem seems to be constantly pegging my CPU. Please see attached screenshots.

Anything I can do to help your team troubleshoot?

Relevant details:

  • Ubuntu 18.04 and Debian on WSL2, but none are running anything at the moment.
  • Windows insider build 18917.1000 on Macbook Air (8 GB, 4 proc).

image
image

@crispinb

This comment has been minimized.

Copy link

@crispinb crispinb commented Jul 2, 2019

After moving to WSL2, VmMem seems to be constantly pegging my CPU. Please see attached screenshots.

I'm experiencing this too. It happens every time I run wsl after some elapsed time (varying from minutes to hours). It continues regardless of what's running in wsl, and I need to issue a wsl --shutdown to stop it.

Windows insider build: 18922.1000
Linux distro: 18.04

I assume this really belongs in a new issue, but wanted to group my comment with that of @mithunshanbhag so I'll leave it here.

@maldo-ctrl

This comment has been minimized.

Copy link

@maldo-ctrl maldo-ctrl commented Jul 22, 2019

Hi,
Same problem here.
Lots of memory consumption with WSL2, nodejs app and VS Code with Remote - WSL while very low consumption in WSL1 with the same tasks.

@liyo

This comment has been minimized.

Copy link

@liyo liyo commented Aug 1, 2019

Same issue on Win build 18950

@zachChilders

This comment has been minimized.

Copy link

@zachChilders zachChilders commented Aug 13, 2019

@benhillis - Do we have a rough ETA on the fix? I'm actually hitting OOM issues on a machine with 32 gigs while just running a normal ninja build that should succeed.

Build 18955

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Aug 13, 2019

@zachChilders - ETA on this one is a bit hard unfortunately. Certainly before WSL2 ships to non-insiders.

@forairan

This comment has been minimized.

Copy link

@forairan forairan commented Aug 19, 2019

Can confirm this on 18963 as well. WSL 2 ate most (29GB+ of 32GB) of my RAM after less than 15 minutes of uptime for no good reason; all I've done is a rsync from the host NTFS volume to the ext4 volume, and an apt-get update.

@strike5150

This comment has been minimized.

Copy link

@strike5150 strike5150 commented Aug 22, 2019

I can confirm the same issue on Microsoft Windows [Version 10.0.18963.1000]

C:\Users\david_d>wsl -l -v
NAME STATE VERSION

  • Ubuntu-18.04 Running 2
    Ubuntu Stopped 1

image

I at the time was running gdb

@rob-solana

This comment has been minimized.

Copy link

@rob-solana rob-solana commented Aug 22, 2019

how about a workaround? this just started happening for me (with 18965) and it has effectively killed my WSL 2 environment

@mbc07

This comment has been minimized.

Copy link

@mbc07 mbc07 commented Aug 22, 2019

Depending of your usage, nocache utility can be used as a workaround to greatly reduce WSL 2 memory consumption, especially if whatever you're running inside it does a lot of filesystem operations...

@rob-solana

This comment has been minimized.

Copy link

@rob-solana rob-solana commented Aug 22, 2019

does nocache follow children? I could start my shell with "nocache"...

@mbc07

This comment has been minimized.

Copy link

@mbc07 mbc07 commented Aug 22, 2019

Haven't tried, but if I understood how nocache works correctly, it should...

@rob-solana

This comment has been minimized.

Copy link

@rob-solana rob-solana commented Aug 22, 2019

thanks! nocache has changed my env from "completely broken" to "merely sucks"

@Beretta1979

This comment has been minimized.

Copy link

@Beretta1979 Beretta1979 commented Aug 27, 2019

thanks! nocache has changed my env from "completely broken" to "merely sucks"

Hi, could you explain how nocache should be used?

@wmulligan87

This comment has been minimized.

Copy link

@wmulligan87 wmulligan87 commented Aug 28, 2019

Hi @benhillis , any update on when a fix for this might be deployed to insiders? My 8gb of ram is really getting hammered by this issue... :(

@rob-solana

This comment has been minimized.

Copy link

@rob-solana rob-solana commented Aug 29, 2019

I wrap compilations in nocache to keep limping along. wrapping nocache around a shell works, but breaks emacs. I've also ordered more RAM :-(

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Aug 29, 2019

Sorry no firm timeline, certainly before WSL2 is out of Insider builds though.

@fabienheureux

This comment has been minimized.

Copy link

@fabienheureux fabienheureux commented Aug 29, 2019

Hi @benhillis , do you have a workaround you could recommend though ?

@fabienheureux

This comment has been minimized.

Copy link

@fabienheureux fabienheureux commented Aug 29, 2019

Well...I need it all day long and the memory consumption goes crazy after only a few minutes (using docker and a quite heavy stack: elastic search, lots of workers, some databases etc). So...not an option unfortunately. Maybe go back to linux while it is not fixed 😎

@weberdominik

This comment has been minimized.

Copy link

@weberdominik weberdominik commented Aug 29, 2019

I am facing the same issue. Is there any workaround (except shutting wsl down) for this @benhillis ?

@microsoft microsoft deleted a comment from maldo-ctrl Aug 29, 2019
@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Aug 29, 2019

Not currently. For some additional context this change requires some changes to the Linux kernel that are in the process of being upstreamed. We will be integrating these changes into the WSL2 kernel as soon as we can.

@apostolos

This comment has been minimized.

Copy link

@apostolos apostolos commented Aug 30, 2019

Workaround: Create a %UserProfile%\.wslconfig file in Windows and use it to limit memory assigned to WSL2 VM.

Example

[wsl2]
memory=6GB
swap=0
localhostForwarding=true

This will still consume the entire 6GBs regardless of Linux memory usage, but at least it'll stop growing more than that.

Supported settings are documented here.

@seasona

This comment has been minimized.

Copy link

@seasona seasona commented Oct 26, 2019

Same problem in 19008.1000, the ram will surge if you compile or open a vscode, but it returns really slow if you close the vscode and so on. I tested that the ram charges from 280m to 1280m through the compile,drops to 1032m after the compile but never get back to 280m again. Fortunately the .wslcofig is really useful

@guaychou

This comment has been minimized.

Copy link

@guaychou guaychou commented Oct 27, 2019

How can i fix this ?

@guaychou

This comment has been minimized.

Copy link

@guaychou guaychou commented Oct 27, 2019

How can i fix this ?

Read above, they said they corrected this error, they only need to be implemented in the next windows update (possibly).

Okay , thank you

@myheartsgoon

This comment has been minimized.

Copy link

@myheartsgoon myheartsgoon commented Oct 28, 2019

@dariothornhill, the best workaround that is working for me is to create a wslconfig file (%UserProfile%\.wslconfig) limiting the amount of RAM of the wsl2 vm and using additional swap space for RAM demanding operations.

[wsl2]
memory=4GB
swap=16GB
localhostForwarding=true

It is running really smooth in a laptop with SSD storage for the swap.

For the changes to the take effect you have to restart WSL 2 vm through powershell or cmd command wsl --shutdown

Thanks a lot, it really saved my 8GB ram laptop. I set ram limit to 512MB, and use 8GB swap as below:

[wsl2]
memory=512MB
swap=8GB
localhostForwarding=true

Before this change, ram usage always spikes to 1.5G+ after I run the docker, but for now, the ram usage is limit to 512MB as Task Explorer shows below and you can also find in machine that wsl2 start to use swap when memory 500MB exceeds.

image
image

@cmeiklejohn

This comment has been minimized.

Copy link

@cmeiklejohn cmeiklejohn commented Oct 29, 2019

Weird, the %userprofile%\.wslconfig still doesn't work for me. I verified the right path and the right line endings. Do you have a blank line at the end of the file, perhaps?

@saikrishnav

This comment has been minimized.

Copy link
Member

@saikrishnav saikrishnav commented Oct 29, 2019

@cmeiklejohn - Can you elaborate? what settings are you using?
It just worked for me as is.

@cmeiklejohn

This comment has been minimized.

Copy link

@cmeiklejohn cmeiklejohn commented Oct 29, 2019

Looks like 19013 fixes this.

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Oct 29, 2019

@cmeiklejohn - Correct, 19013 has the fix for this.

@desperadoshi

This comment has been minimized.

Copy link

@desperadoshi desperadoshi commented Oct 30, 2019

My Windows 10 Build 19013.vb_release.191025-1609 is still bothered by this issue. WSL2 Ubuntu 18.04 LTS.

ver

I'm using VSCode remotely editing codes in WSL2. Vmmem consumes large memory and does not release them.

mem

I close VSCode without executing wsl --shutdown. Vmmem does not release the memory.

mem2

Only after wsl --shutdown, the memory is given back to Windows.

@peter-developer

This comment has been minimized.

Copy link

@peter-developer peter-developer commented Oct 30, 2019

My Windows 10 Build 19013.vb_release.191025-1609 is still bothered by this issue. WSL2 Ubuntu 18.04 LTS.
mem
ver

exec free -h the safest is that the cache memory is occupying ram memory, free the cache and see if the memory returns to windows (before this did not work)

@desperadoshi

This comment has been minimized.

Copy link

@desperadoshi desperadoshi commented Oct 30, 2019

My Windows 10 Build 19013.vb_release.191025-1609 is still bothered by this issue. WSL2 Ubuntu 18.04 LTS.
mem
ver

exec free -h the safest is that the cache memory is occupying ram memory, free the cache and see if the memory returns to windows (before this did not work)

free -h shows that buff/cache is not released.

As a try, I copied a folder of large size to another place. And this copy operation leads the buff/cache to increase a lot. When the copy is finished, the buff/cache is not released. So is Vmmem. Check the following pic.

mem_free

@fanvinga

This comment has been minimized.

Copy link

@fanvinga fanvinga commented Oct 30, 2019

My Windows 10 Build 19013.vb_release.191025-1609 is still bothered by this issue. WSL2 Ubuntu 18.04 LTS.
mem
ver

exec free -h the safest is that the cache memory is occupying ram memory, free the cache and see if the memory returns to windows (before this did not work)

free -h shows that buff/cache is not released.

As a try, I copied a folder of large size to another place. And this copy operation leads the buff/cache to increase a lot. When the copy is finished, the buff/cache is not released. So is Vmmem. Check the following pic.

mem_free

You may need such command like sync; echo 3 > /proc/sys/vm/drop_caches to purge Linux cache.

@desperadoshi

This comment has been minimized.

Copy link

@desperadoshi desperadoshi commented Oct 30, 2019

My Windows 10 Build 19013.vb_release.191025-1609 is still bothered by this issue. WSL2 Ubuntu 18.04 LTS.
mem
ver

exec free -h the safest is that the cache memory is occupying ram memory, free the cache and see if the memory returns to windows (before this did not work)

free -h shows that buff/cache is not released.
As a try, I copied a folder of large size to another place. And this copy operation leads the buff/cache to increase a lot. When the copy is finished, the buff/cache is not released. So is Vmmem. Check the following pic.
mem_free

You may need such command like sync; echo 3 > /proc/sys/vm/drop_caches to purge Linux cache.

OK. It works.

@bobotu

This comment has been minimized.

Copy link

@bobotu bobotu commented Oct 30, 2019

Can WSL2 automatically drop some cache when the Windows host has pressure on memory?

@peter-developer

This comment has been minimized.

Copy link

@peter-developer peter-developer commented Oct 30, 2019

I want to report another behavior related to VSCode. I use VS Code to remotely do code development. I'm not so sure if this issue should be reported here or not.

In Windows 10 Build 19013.vb_release.191025-1609, VS Code version 1.39.2 with Remote-WSL plugin 0.39.9, doing search in VS Code leads Vmmem to consume large memory. And I need to manually release the cache by sudo sh -c "/bin/echo 3 > /proc/sys/vm/drop_caches". However, such behavior is really annoying. Why does searching in VS Code give such results?

Or should I report this to VS Code team?

vscode

Linux uses RAM as a cache when writing to disk, this to gain speed. It is a usual kernel behavior and yes, you can control this behavior. Research Linux and ways to control the cache. The comments above talk about how to mitigate that behavior automatically.

@desperadoshi

This comment has been minimized.

Copy link

@desperadoshi desperadoshi commented Oct 30, 2019

Can WSL2 automatically drop some cache when the Windows host has pressure on memory?

I feel like it is. But I'm not sure. In a short period of time like 1 minute, I cannot feel the decrease. But it seems like it drops the RAM after some time.

@craigloewen-msft

This comment has been minimized.

Copy link
Member

@craigloewen-msft craigloewen-msft commented Oct 30, 2019

We've released a blog post to help go over the details of this feature: Memory Reclaim in the Windows Subsystem for Linux 2. It should help clear up how memory becomes freed!

@i2

This comment has been minimized.

Copy link

@i2 i2 commented Oct 31, 2019

I created a .wslconfig file in %UserProfile% but it doesn't take affect. I test it by setting the following:

[wsl2]
swap=0
processors=3

but when I do 'top' it still shows values from before this change. What am I missing?

@ErvalhouS

This comment has been minimized.

Copy link

@ErvalhouS ErvalhouS commented Oct 31, 2019

Great, now the issue is just about buff/cache not being released.

@Indribell

This comment has been minimized.

Copy link

@Indribell Indribell commented Nov 1, 2019

Great, now the issue is just about buff/cache not being released.

Its a issue that is being a bit overlooked. Linux really does not have the habit of freeing memory from it cache/buffer and like to hold everything in memory forever ( only pushing out things that are not accessed in a long time, to make space for new things to cache ). So its a valid point, that is unfortunately somewhat ignored.

I think the expectations of the MS team is: You are done with your task, you close the bash and it auto frees memory. But if you have VSC open and you put your PC to sleep every day ( more productive then reopening everything every day ), it never gets freed. Thus it keep growing as you compile more code or do other file operations. And you are forced to put memory limits on those WSL instances.

By default WSL2 needs to have some kind of a limit or force the Linux kernel to be more aggressive in reclaiming the cache. For example: everything in cache, that is not used in 1 hour time, reclaim it ( a setting you can also override per WSL2 instance ).

/Edit:

And it becomes a issue fast when you work with a lot of microservices / have a lot of WSL2 instances open / like to have clean WSL2 instances by having your data cleanly separated per WSL2 /VM instance.

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Nov 1, 2019

@Indribell - we are looking at improving the behavior in the future to improve the behavior regarding Linux's cached memory.

@ErvalhouS

This comment has been minimized.

Copy link

@ErvalhouS ErvalhouS commented Nov 1, 2019

Wouldn't something like setting up a line at sudo crontab -e with the following do the desired effect?

0 * * * * echo 3 > /proc/sys/vm/drop_caches

I don't know the implication of that in the performance of the machine, but since it's not meant to be used in a production environment I think clearing memory cache every hour seems to be a nice way to workaround that issue.

@benhillis

This comment has been minimized.

Copy link
Member

@benhillis benhillis commented Nov 1, 2019

@ErvalhouS - The performance impacts are substantial. When you drop the page cache you're going to be reading from disk so it's not something we want to do without being sure you're done with the files that are cached.

@discotroll65

This comment has been minimized.

Copy link

@discotroll65 discotroll65 commented Nov 6, 2019

@benhillis et all, I'm a relative noob to the inner workings of the linux kernel, but might some way of addressing this be making a whitelist of processes whose memory affects can be cleared after they are done? For example, I unzip a .sql.gz file in my WSL instance, and now it permanently has an extra gig of memory tied up to the instance. Would there be an easy way to release from the cache those unzipped files?

@romitzz1

This comment has been minimized.

Copy link

@romitzz1 romitzz1 commented Nov 8, 2019

This might be closer to a Docker issue, but I have been using the edge Docker for Windows with WSL 2. In my case, the `buff/cache' can take 12GB during the building process, and once everything is running it is using 20GB before dropping the cache.
Where as running purely in the old Hyper V version, I had no issues leaving the environment at 12GB max.

@nisarg-ujjainkar

This comment has been minimized.

Copy link

@nisarg-ujjainkar nisarg-ujjainkar commented Nov 9, 2019

Can't run the command "sudo sync; echo 3 > /proc/sys/vm/drop_caches". It says permission denied

@peter-developer

This comment has been minimized.

Copy link

@peter-developer peter-developer commented Nov 9, 2019

Can't run the command "sudo sync; echo 3 > /proc/sys/vm/drop_caches". It says permission denied
1- sudo su
2- sync; echo 3 > /proc/sys/vm/drop_caches

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.