Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

vpnkit.exe eating my RAM #371

Open
soar opened this issue Mar 25, 2018 · 83 comments

Comments

@soar
Copy link

commented Mar 25, 2018

This happened after updating to 18.* branch. At this moment I have 18.03.0-ce-rc4-win57 (16511) build and my vpnkit.exe steals gigabytes of RAM in 2-3 hours with 2 containers running.

Like this:

2018-03-25 18-00-20

Or even like this:

2018-03-24 16-10-41

I think it should never consume about 8 GBs of my RAM.

@bartoncasey

This comment has been minimized.

Copy link

commented Mar 26, 2018

I have this issue as well. vpnkit.exe claims memory in proportion to the amount of network activity into and out of the docker containers, and never releases it.

Edit: the latest version 18.03.0-ce-win58 (16761) may have resolved the issue. Fingers crossed.

@soar

This comment has been minimized.

Copy link
Author

commented Mar 26, 2018

I've updated my Docker to 18.03.0-ce-win58 (16761) two hours ago - and problem is still here:

image

@bartoncasey

This comment has been minimized.

Copy link

commented Mar 27, 2018

1 day later it's sitting at 1.2G, from light traffic.

@laarmen

This comment has been minimized.

Copy link

commented Mar 29, 2018

We are several here at work having this exact issue.

@djs55

This comment has been minimized.

Copy link
Collaborator

commented Mar 29, 2018

Thanks for your reports.

In order to make progress with this issue I need some reproduction steps. Could you provide a docker-compose.yml (or similar) and instructions to reproduce the problem?

@laarmen

This comment has been minimized.

Copy link

commented Mar 30, 2018

After trying a bit, it seems opening an HTTPS connection to a server on our internal network triggers the bug. The same doesn't apply to external, public servers (i.e. docker.com) nor other Docker instances.

@djs55

This comment has been minimized.

Copy link
Collaborator

commented Mar 30, 2018

@laarmen thanks for the update. Could you trigger the bug and then upload a diagnostic report? I'd like to take a look at the logs.

@laarmen

This comment has been minimized.

Copy link

commented Mar 30, 2018

See https://github.com/laarmen/VpnKitPoC for the code. How can I do the diagnostic report thing?

@djs55

This comment has been minimized.

Copy link
Collaborator

commented Mar 30, 2018

@laarmen

This comment has been minimized.

Copy link

commented Mar 30, 2018

I'm not entirely sure this was an instance of the bug as I was still under the 300MB bar of RAM used by vpnkit, but it was consistently climbing. I'll upload another report if I get to the "eat-my-RAM" levels later.

ID: 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_18-33-45

@laarmen

This comment has been minimized.

Copy link

commented Mar 30, 2018

In case that's useful, I just stopped all the docker containers on my workstation, waited a few minutes, and the vpnkit process sits at 700MB. I uploaded a second diagnosis, see 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_19-09-21

@laarmen

This comment has been minimized.

Copy link

commented Mar 30, 2018

This time on my home computer and network, same code except that the target (on local network) is using plain http (no SSL), the memory grew to 1.5GB.

ID: D25DA2F3-2F67-42BA-A292-78A39BCBAEC4/2018-03-30_20-34-58

@cnuernber

This comment has been minimized.

Copy link

commented Apr 1, 2018

We are having same issue. This is happening to us in under one day (although we are using an app that generates a lot of network traffic). So we currently have to bounce docker once per day.

pastedimage

@krukid

This comment has been minimized.

Copy link

commented Apr 3, 2018

Same thing for me - I'm running a single Node.js process that downloads files from the web over HTTP (text and binary) - some 25-30K files, ~1GB in volume, about 100KB/s. VPNKit process consumes all available RAM within hours (I've had it consume up to 9GB of RAM, even though the overall limit for Docker itself is 2GB).

@bazzilic

This comment has been minimized.

Copy link

commented Apr 4, 2018

Same here. Win 10 x64, docker version:

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:28 2018
 OS/Arch:       windows/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.24)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:21:06 2018
  OS/Arch:      windows/amd64
  Experimental: true

vpnkit.exe currently at 7GB, constantly climbing:

image

0 containers running. Uploaded a diagnostic, id: 29542F91-6441-4210-934F-DB948F4EF0EF/2018-04-04_16-34-22

UPDATE 24/04/2018:
As suggested below, adding vpnkit commit sha:

bazzilic@CSLRF21 $ & "C:\Program Files\Docker\Docker\resources\vpnkit.exe" --debug --ethernet foo
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is 7c425f691978cb4a708ccc295dd331eae5cebc85
@dsmaher

This comment has been minimized.

Copy link

commented Apr 5, 2018

Here's a simple docker-compose.yaml that can reproduce the issue. If you watch memory usage on vpnkit.exe when this is running, it climbs by almost 1M every time the wget runs.

version: '3'
services:
  eat-memory:
    image: busybox
    entrypoint: sh
    command:
      - -c
      - |
        while true; do
          echo Getting docker.com...
          wget -qO/dev/null https://www.docker.com
          sleep 5
        done
@bazzilic

This comment has been minimized.

Copy link

commented Apr 5, 2018

In my case it climbs even if there’s no activity related to docker at all. At least, nothing explicit.

@pbering

This comment has been minimized.

Copy link

commented Apr 9, 2018

Even with all windows and linux containers stopped, the memory usage is constantly around 1.5 GB on my machine...

@logich

This comment has been minimized.

Copy link

commented Apr 13, 2018

I have this same issue occurring with 3 containers that are doing a large amount of WAN activity. If left to run over a week this will consume all the available RAM and leave the system in a unstable state. My only work around is restarting docker regularly.

@tcederquist

This comment has been minimized.

Copy link

commented Apr 20, 2018

Cross link forum entry of many folks with the same vpnkit memory issue: https://forums.docker.com/t/vpnkit-uses-all-free-memory/48558/12 For me I suspect the behavior appeared with the 16762 build - never noticed this before but wasn't looking until it exhausted my memory for one simple nginx container.

@arnie311

This comment has been minimized.

Copy link

commented Apr 20, 2018

I am having the same issue.
Docker Version 18.03.0-ce-win59 (16762)
Windows Server 2016 with 32GB of Memory
Limit docker to 10gb of memory and vpnkit.exe uses up to 16GB in 24 hours.
snag_289fdd8a
It either crashes or I have to restart Docker

@jtownley

This comment has been minimized.

Copy link

commented Apr 23, 2018

I hate to post a me too but, me too:
Docker Version 18.03.0-ce-mac60 (23751)
Channel: stable
6ddfc0f1d3
OSX 10.11.6 (16GB Ram)
Running one talkative (http outgoing requests only) app via docker compose

@omnipitous

This comment has been minimized.

Copy link

commented Apr 23, 2018

"me too" Left a couple (mostly idle) containers running over the weekend came back to 4GB used by vpnkit and a cranky system as that's what I had left..

Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:06:28 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:14:32 2018
OS/Arch: linux/amd64
Experimental: false

@tsasioglu

This comment has been minimized.

Copy link

commented Apr 23, 2018

We've rolled back to 17.12.0-ce-win47 2018-01-12 and are no longer seeing this issue.

@bartoncasey

This comment has been minimized.

Copy link

commented Apr 23, 2018

Had an idea to compare versions so that we can isolate the vpnkit version that started failing.

PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --version
%%VERSION%%
PS C:\Program Files\Docker\Docker\Resources>

Sigh.

@bartoncasey

This comment has been minimized.

Copy link

commented Apr 23, 2018

.\vpnkit.exe --debug --ethernet foo will spit out a git sha:

PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --debug --ethernet xxx
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is eb91fd8319abdfcaf87a1839e46b7ce0577b68fc
...

That's for the current Version 18.04.0-ce-rc2-win61 (17070). It corresponds to the most recent commit here.

@tsasioglu What does your 17.12.0-ce-win47 report?

@imarotte

This comment has been minimized.

Copy link

commented Apr 27, 2018

Is there an easy way to roll back to 17.12.0-ce-win? As it is, 18 is completely unusable for me. I have to restart every 90 min because vpnkit uses +90% of my memory

@apm963

This comment has been minimized.

Copy link

commented May 18, 2018

The provided Windows vpnkit.exe binary fixes the issue for me. I have been hovering between 8 and 23 MB for the past few hours with the same networking conditions as prior tests. Thank you very much for finding and patching this!

Edit: 5 hours later and the memory usage is still very low - currently at 34 MB which is still very reasonable. I'm comfortable leaving this running over the weekend. As far as I'm concerned the biggest issue has been fixed.

2nd edit: It is at ~95 MB after 70 hours uptime and constant network IO. Still within very comfortable levels. Thanks again!

@ericblade

This comment has been minimized.

Copy link

commented May 18, 2018

trying that idea out, i ran with a fresh restart of the 18.05 vpnkit, and just hammered my node server by holding down the "submit form" button on a page, and was able to get it from 18MB to 500MB in about 10 seconds. I replaced the vpnkit with the one linked above, and it went from 12MB to 26MB in about the same time. Definite improvement. :-D It has relaxed back to 23MB in the couple of minutes since then.

@ericblade

This comment has been minimized.

Copy link

commented May 19, 2018

After 5 hours it is nice and comfortable at 15.8MB it looks like. 🥂

@Thunderchickenz

This comment has been minimized.

Copy link

commented May 19, 2018

Yep after 8 hours with the new vpnkit on Windows things are stable and I'm not seeing any problems. Thanks!

@ericblade

This comment has been minimized.

Copy link

commented May 19, 2018

I've now been holding between 27 and 32MB for the last several hours, it seems (have been glancing at it occasionally, not really following it). Looks like it's still "leaking", though it's lacking the massive allocations that were occuring before, so it doesn't look near as bad. I mean, a 15-ish MB leak in about a day isn't exactly something that freaks me out, but it's also something that I'd definitely want to ferret out and take care of. What I don't know is if it will eventually return to the 15-ish MB base, or if it's going to raise further. Guess we'll see in another day or two. :-D

@ericblade

This comment has been minimized.

Copy link

commented May 22, 2018

2.5 days, holding at 48MB.

@ericblade

This comment has been minimized.

Copy link

commented May 22, 2018

Was up to 52MB today. My nginx container had stopped responding to any requests, however. I used docker-compose restart nginx, and vpnkit immediately dropped back to 16MB, and the new container started responding to requests. Not sure if it's some kind of interaction between nginx and docker, nginx and my node server (possibly including other web servers inside docker), or what.

@welja

This comment has been minimized.

Copy link

commented Jun 13, 2018

@djs55 Just tried your fix (windows10) and the memory went from 1.2 GB to 13 MB
Thanks!

@ericblade

This comment has been minimized.

Copy link

commented Jun 13, 2018

I see this fix made it to the latest update, so that's good. I can update in the next day or two.

FWIW, I don't even know how many days it's been since I last restarted things, and vpnkit is sitting at 58MB usage right now.

So, unless there's been additional fixes since the replacement exe above, it's still leaking.. it's just not severe.

@johannes69ki

This comment has been minimized.

Copy link

commented Aug 2, 2018

FYI: The current edge release 18.06.0-ce-win71 (19101) still contains the buggy vpnkit.exe. After replacing the original binary with the one posted here, I could reduce my memory footprint from ~6GB to 180MB which is IMHO still too much for what this process presumably does but at least now I can get some work/testing done on my machine...

@apm963

This comment has been minimized.

Copy link

commented Aug 2, 2018

Odd, I updated to stable last week (18.06.0-ce-win72 (19098), prior to that I was running Edge with the linked binary) and am seeing the same behavior as the binary linked above, which is good. Docker has been running for 6 days and my vpnkit.exe process is hovering between 300 - 400MB. This is still high, but it would have maxed out my RAM within a few hours of starting Docker if the severe bug reported in this thread still existed.

As a side note I believe I read a comment in this thread where there was another slow leak identified but I may have been misinterpreting it.

@johannes69ki

This comment has been minimized.

Copy link

commented Aug 2, 2018

Hey @apm963, thanks for your reply. I now realize that I might be wrong with my above accusation... I am sorry! But now I realize what is really going on:

I manually pause the MobyLinuxVM in Hyper-V-Manager whenever I am not actively using docker/kubernetes. Whenever I pause the VM you can watch vpnkit.exe to accumulate RAM. After I unpause the VM, then vpnkit.exe does not accumulate RAM anylonger (at least not at such a pace).

The reason why I am pausing the VM is because I am working with kubernetes and my machine is a laptop. I realized that the vmmem.exe process drains my battery since it constantly consumes about 18 percent of my CPU, even if I haven't any pods running in kubernetes. (The vmmem.exe process shows the combined resource usage of all non-host VMs currently running on your Windows: https://blogs.msdn.microsoft.com/oldnewthing/20180717-00/?p=99265) So, in order to save battery I pause the VM whenever I am not actively working with docker/kubernetes.

I am new to kubernetes, but here it says, that just running all the management related processes of kubernetes itself causes about 95 percent of single core CPU usage (i.e. the minimum requirement for kubernetes is a two core CPU; you might want to have a look into the "CPU and Memory Requirements" section here https://www.weave.works/docs/net/latest/kubernetes/kube-addon/). My laptop has a quad core CPU, so 18 percent vmmem.exe CPU usage matches exactly that 95 percent single core figure...

I think that the memory issue with vpnkit.exe also happens whenever my laptop goes into energy saving mode... So, the issue not only comes up whenever I manually pause the VM but also when my laptop suspends itself into an energy saving mode when the VM is paused automatically but vpnkit.exe is still active in the background.

@ericblade

This comment has been minimized.

Copy link

commented Aug 3, 2018

hmmm. I had just upgraded to the latest Edge version, 18.06-0 . . . on the previous version, when i attempted to shut down Docker to upgrade, everything got all screwed up, and vmmem.exe skyrocketed the CPU to peg it at 90%, and Docker refused to respond to anything, and I had to reboot the computer before I could upgrade.

I just hit the web server (the main process doesn't run in Docker, but there's a nginx container that is gating to it) about a dozen times, and every page load increased the memory usage on vpnkit by between 500 and 600K.

It does look like there's a problem again. I had been sitting around 15MB usage consistently the last time I looked, which was whatever version included the change above.

@pit

This comment has been minimized.

Copy link

commented Oct 17, 2018

Are there any plans to move this fix into Docker Mac official release?
Version 18.06.1-ce-mac73 (26764) - memory leak still present

@michaeljon

This comment has been minimized.

Copy link

commented Oct 24, 2018

I'm on 2.0.0.0-beta1-mac75 (27117) and am still seeing this. Are we still waiting on a fix to make it into one of the release branches?

@jasonrhaas

This comment has been minimized.

Copy link

commented Oct 24, 2018

I'm on 2.0.0.0-beta1-mac75 (27117) and am still seeing this. Are we still waiting on a fix to make it into one of the release branches?

I'm not seeing large memory using for vpnkit.exe, but my docker-for-mac seems to have a significant memory leak. I starts off at 6 GB and keeps climbing up to about 20 GB after it has been running for a while.

@MichaelLeeHobbs

This comment has been minimized.

Copy link

commented Dec 18, 2018

Network heavy projects. We only using Docker for Windows on our Dev machines but we have to reboot at least once a day as even with Docker restarts the vpnkit memory issue seems to leave the machines unstable and running poorly.

@soar

This comment has been minimized.

Copy link
Author

commented Jan 8, 2019

I still experience this issue on Docker 2.0.0.0-mac81

@00Asgaroth00

This comment has been minimized.

Copy link

commented Jan 22, 2019

I have the same issue using docker for windows version 2.0.0.2 (30215) build 0b030e1. I'm on Windows 10 Pro. If you need any additional info please let me know.

@MichaelLeeHobbs

This comment has been minimized.

Copy link

commented Feb 18, 2019

vpnkit.exe seems to no longer be causing us issue on our network heavy projects. For the most part, Docker on Windows seems way more stable than back in December.

@Ciantic

This comment has been minimized.

Copy link

commented Feb 21, 2019

I don't have any containers running, yet the vpnkit.exe takes ~2.4 Gb of memory.

Version 2.0.0.3 (31259), Channel: stable, Build: 8858db3

@TomYeoman

This comment has been minimized.

Copy link

commented Mar 25, 2019

Left container on overnight and had 5gb RAM being used + heavy machine freezes by the morning after doing various network requests within it overnight

@WilliamStone

This comment has been minimized.

Copy link

commented Jun 28, 2019

I don't have any containers running, yet the vpnkit.exe takes a lot of memory: 5.7 GB of working set and 9.5 GB of commit size. Computer total memory is 32GB.
Every time kill vpnkit.exe, and it automatically re-runs and allocates 9.5 GB of commit size in 1 minutes.

System: Windows10
Version 2.0.0.3 Build 31259 (Stable) & 2.0.5.0 Build 35318 (Edge) [Latest version]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
You can’t perform that action at this time.