vpnkit.exe eating my RAM

[soar]

This happened after updating to 18.* branch. At this moment I have 18.03.0-ce-rc4-win57 (16511) build and my vpnkit.exe steals gigabytes of RAM in 2-3 hours with 2 containers running.

Like this:

Or even like this:

I think it should never consume about 8 GBs of my RAM.

[bartoncasey]

I have this issue as well. vpnkit.exe claims memory in proportion to the amount of network activity into and out of the docker containers, and never releases it.

Edit: the latest version 18.03.0-ce-win58 (16761) may have resolved the issue. Fingers crossed.

[soar]

I've updated my Docker to 18.03.0-ce-win58 (16761) two hours ago - and problem is still here:

[bartoncasey]

1 day later it's sitting at 1.2G, from light traffic.

[laarmen]

We are several here at work having this exact issue.

[djs55]

Thanks for your reports.

In order to make progress with this issue I need some reproduction steps. Could you provide a docker-compose.yml (or similar) and instructions to reproduce the problem?

[laarmen]

After trying a bit, it seems opening an HTTPS connection to a server on our internal network triggers the bug. The same doesn't apply to external, public servers (i.e. docker.com) nor other Docker instances.

[djs55]

@laarmen thanks for the update. Could you trigger the bug and then upload a diagnostic report? I'd like to take a look at the logs.

[laarmen]

See https://github.com/laarmen/VpnKitPoC for the code. How can I do the diagnostic report thing?

[djs55]

On Windows there should be a whale-shaped icon in the system tray. After right clicking on it there should be a menu item called something like "Diagnose and Feedback". Clicking on this should take you to a dialog where diagnostics are uploaded and assigned a unique id. If you quote the id in the ticket then I can download the logs and take a look. (Sorry I couldn't give more precise instructions but I don't have a Windows machine to hand)

…

On Fri, Mar 30, 2018 at 2:19 PM, Simon Chopin ***@***.***> wrote: See https://github.com/laarmen/VpnKitPoC for the code. How can I do the diagnostic report thing? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#371 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAMHul2HIQrFUI_nFNH_aLxIOT-jhgCTks5tjjDlgaJpZM4S6LDK> .

[laarmen]

I'm not entirely sure this was an instance of the bug as I was still under the 300MB bar of RAM used by vpnkit, but it was consistently climbing. I'll upload another report if I get to the "eat-my-RAM" levels later.

ID: 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_18-33-45

[laarmen]

In case that's useful, I just stopped all the docker containers on my workstation, waited a few minutes, and the vpnkit process sits at 700MB. I uploaded a second diagnosis, see 5E3BFA7A-FF8F-4077-8583-773FF79518CC/2018-03-30_19-09-21

[laarmen]

This time on my home computer and network, same code except that the target (on local network) is using plain http (no SSL), the memory grew to 1.5GB.

ID: D25DA2F3-2F67-42BA-A292-78A39BCBAEC4/2018-03-30_20-34-58

[cnuernber]

We are having same issue. This is happening to us in under one day (although we are using an app that generates a lot of network traffic). So we currently have to bounce docker once per day.

[krukid]

Same thing for me - I'm running a single Node.js process that downloads files from the web over HTTP (text and binary) - some 25-30K files, ~1GB in volume, about 100KB/s. VPNKit process consumes all available RAM within hours (I've had it consume up to 9GB of RAM, even though the overall limit for Docker itself is 2GB).

[bazzilic]

Same here. Win 10 x64, docker version:

Client:
 Version:       18.03.0-ce
 API version:   1.37
 Go version:    go1.9.4
 Git commit:    0520e24
 Built: Wed Mar 21 23:06:28 2018
 OS/Arch:       windows/amd64
 Experimental:  false
 Orchestrator:  swarm

Server:
 Engine:
  Version:      18.03.0-ce
  API version:  1.37 (minimum version 1.24)
  Go version:   go1.9.4
  Git commit:   0520e24
  Built:        Wed Mar 21 23:21:06 2018
  OS/Arch:      windows/amd64
  Experimental: true

vpnkit.exe currently at 7GB, constantly climbing:

0 containers running. Uploaded a diagnostic, id: 29542F91-6441-4210-934F-DB948F4EF0EF/2018-04-04_16-34-22

UPDATE 24/04/2018:
As suggested below, adding vpnkit commit sha:

bazzilic@CSLRF21 $ & "C:\Program Files\Docker\Docker\resources\vpnkit.exe" --debug --ethernet foo
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is 7c425f691978cb4a708ccc295dd331eae5cebc85

[dsmaher]

Here's a simple docker-compose.yaml that can reproduce the issue. If you watch memory usage on vpnkit.exe when this is running, it climbs by almost 1M every time the wget runs.

version: '3'
services:
  eat-memory:
    image: busybox
    entrypoint: sh
    command:
      - -c
      - |
        while true; do
          echo Getting docker.com...
          wget -qO/dev/null https://www.docker.com
          sleep 5
        done

[bazzilic]

In my case it climbs even if there’s no activity related to docker at all. At least, nothing explicit.

[pbering]

Even with all windows and linux containers stopped, the memory usage is constantly around 1.5 GB on my machine...

[logich]

I have this same issue occurring with 3 containers that are doing a large amount of WAN activity. If left to run over a week this will consume all the available RAM and leave the system in a unstable state. My only work around is restarting docker regularly.

[tcederquist]

Cross link forum entry of many folks with the same vpnkit memory issue: https://forums.docker.com/t/vpnkit-uses-all-free-memory/48558/12 For me I suspect the behavior appeared with the 16762 build - never noticed this before but wasn't looking until it exhausted my memory for one simple nginx container.

[arnie311]

I am having the same issue.
Docker Version 18.03.0-ce-win59 (16762)
Windows Server 2016 with 32GB of Memory
Limit docker to 10gb of memory and vpnkit.exe uses up to 16GB in 24 hours.

It either crashes or I have to restart Docker

[jtownley]

I hate to post a me too but, me too:
Docker Version 18.03.0-ce-mac60 (23751)
Channel: stable
6ddfc0f1d3
OSX 10.11.6 (16GB Ram)
Running one talkative (http outgoing requests only) app via docker compose

[omnipitous]

"me too" Left a couple (mostly idle) containers running over the weekend came back to 4GB used by vpnkit and a cranky system as that's what I had left..

Client:
Version: 18.03.0-ce
API version: 1.37
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:06:28 2018
OS/Arch: windows/amd64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.0-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.4
Git commit: 0520e24
Built: Wed Mar 21 23:14:32 2018
OS/Arch: linux/amd64
Experimental: false

[tsasioglu]

We've rolled back to 17.12.0-ce-win47 2018-01-12 and are no longer seeing this issue.

[bartoncasey]

Had an idea to compare versions so that we can isolate the vpnkit version that started failing.

PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --version
%%VERSION%%
PS C:\Program Files\Docker\Docker\Resources>

Sigh.

[bartoncasey]

.\vpnkit.exe --debug --ethernet foo will spit out a git sha:

PS C:\Program Files\Docker\Docker\Resources> .\vpnkit.exe --debug --ethernet xxx
vpnkit.exe: [INFO] Setting handler to ignore all SIGPIPE signals
vpnkit.exe: [INFO] Version is eb91fd8319abdfcaf87a1839e46b7ce0577b68fc
...

That's for the current Version 18.04.0-ce-rc2-win61 (17070). It corresponds to the most recent commit here.

@tsasioglu What does your 17.12.0-ce-win47 report?

[imarotte]

Is there an easy way to roll back to 17.12.0-ce-win? As it is, 18 is completely unusable for me. I have to restart every 90 min because vpnkit uses +90% of my memory

[ericblade]

2.5 days, holding at 48MB.

[ericblade]

Was up to 52MB today. My nginx container had stopped responding to any requests, however. I used docker-compose restart nginx, and vpnkit immediately dropped back to 16MB, and the new container started responding to requests. Not sure if it's some kind of interaction between nginx and docker, nginx and my node server (possibly including other web servers inside docker), or what.

[welja]

@djs55 Just tried your fix (windows10) and the memory went from 1.2 GB to 13 MB
Thanks!

[ericblade]

I see this fix made it to the latest update, so that's good. I can update in the next day or two.

FWIW, I don't even know how many days it's been since I last restarted things, and vpnkit is sitting at 58MB usage right now.

So, unless there's been additional fixes since the replacement exe above, it's still leaking.. it's just not severe.

[maschinensehen]

FYI: The current edge release 18.06.0-ce-win71 (19101) still contains the buggy vpnkit.exe. After replacing the original binary with the one posted here, I could reduce my memory footprint from ~6GB to 180MB which is IMHO still too much for what this process presumably does but at least now I can get some work/testing done on my machine...

[apm963]

Odd, I updated to stable last week (18.06.0-ce-win72 (19098), prior to that I was running Edge with the linked binary) and am seeing the same behavior as the binary linked above, which is good. Docker has been running for 6 days and my vpnkit.exe process is hovering between 300 - 400MB. This is still high, but it would have maxed out my RAM within a few hours of starting Docker if the severe bug reported in this thread still existed.

As a side note I believe I read a comment in this thread where there was another slow leak identified but I may have been misinterpreting it.

[maschinensehen]

Hey @apm963, thanks for your reply. I now realize that I might be wrong with my above accusation... I am sorry! But now I realize what is really going on:

I manually pause the MobyLinuxVM in Hyper-V-Manager whenever I am not actively using docker/kubernetes. Whenever I pause the VM you can watch vpnkit.exe to accumulate RAM. After I unpause the VM, then vpnkit.exe does not accumulate RAM anylonger (at least not at such a pace).

The reason why I am pausing the VM is because I am working with kubernetes and my machine is a laptop. I realized that the vmmem.exe process drains my battery since it constantly consumes about 18 percent of my CPU, even if I haven't any pods running in kubernetes. (The vmmem.exe process shows the combined resource usage of all non-host VMs currently running on your Windows: https://blogs.msdn.microsoft.com/oldnewthing/20180717-00/?p=99265) So, in order to save battery I pause the VM whenever I am not actively working with docker/kubernetes.

I am new to kubernetes, but here it says, that just running all the management related processes of kubernetes itself causes about 95 percent of single core CPU usage (i.e. the minimum requirement for kubernetes is a two core CPU; you might want to have a look into the "CPU and Memory Requirements" section here https://www.weave.works/docs/net/latest/kubernetes/kube-addon/). My laptop has a quad core CPU, so 18 percent vmmem.exe CPU usage matches exactly that 95 percent single core figure...

I think that the memory issue with vpnkit.exe also happens whenever my laptop goes into energy saving mode... So, the issue not only comes up whenever I manually pause the VM but also when my laptop suspends itself into an energy saving mode when the VM is paused automatically but vpnkit.exe is still active in the background.

[ericblade]

hmmm. I had just upgraded to the latest Edge version, 18.06-0 . . . on the previous version, when i attempted to shut down Docker to upgrade, everything got all screwed up, and vmmem.exe skyrocketed the CPU to peg it at 90%, and Docker refused to respond to anything, and I had to reboot the computer before I could upgrade.

I just hit the web server (the main process doesn't run in Docker, but there's a nginx container that is gating to it) about a dozen times, and every page load increased the memory usage on vpnkit by between 500 and 600K.

It does look like there's a problem again. I had been sitting around 15MB usage consistently the last time I looked, which was whatever version included the change above.

[pit]

Are there any plans to move this fix into Docker Mac official release?
Version 18.06.1-ce-mac73 (26764) - memory leak still present

[michaeljon]

I'm on 2.0.0.0-beta1-mac75 (27117) and am still seeing this. Are we still waiting on a fix to make it into one of the release branches?

[jasonrhaas]

I'm on 2.0.0.0-beta1-mac75 (27117) and am still seeing this. Are we still waiting on a fix to make it into one of the release branches?

I'm not seeing large memory using for vpnkit.exe, but my docker-for-mac seems to have a significant memory leak. I starts off at 6 GB and keeps climbing up to about 20 GB after it has been running for a while.

[MichaelLeeHobbs]

Network heavy projects. We only using Docker for Windows on our Dev machines but we have to reboot at least once a day as even with Docker restarts the vpnkit memory issue seems to leave the machines unstable and running poorly.

[soar]

I still experience this issue on Docker 2.0.0.0-mac81

[00Asgaroth00]

I have the same issue using docker for windows version 2.0.0.2 (30215) build 0b030e1. I'm on Windows 10 Pro. If you need any additional info please let me know.

[MichaelLeeHobbs]

vpnkit.exe seems to no longer be causing us issue on our network heavy projects. For the most part, Docker on Windows seems way more stable than back in December.

[Ciantic]

I don't have any containers running, yet the vpnkit.exe takes ~2.4 Gb of memory.

Version 2.0.0.3 (31259), Channel: stable, Build: 8858db3

[TomYeoman]

Left container on overnight and had 5gb RAM being used + heavy machine freezes by the morning after doing various network requests within it overnight

[WilliamStone]

I don't have any containers running, yet the vpnkit.exe takes a lot of memory: 5.7 GB of working set and 9.5 GB of commit size. Computer total memory is 32GB.
Every time kill vpnkit.exe, and it automatically re-runs and allocates 9.5 GB of commit size in 1 minutes.

System: Windows10
Version 2.0.0.3 Build 31259 (Stable) & 2.0.5.0 Build 35318 (Edge) [Latest version]

[PeshekhonovK]

Hi everyone, is there any fix for this issue? Experiencing it on win10, probably latest stable Docker Desktop...

[ZILosoft]

same issue

[innomaxx]

same issue
Docker version 20.10.12, build e91ed57

[mykola-dev]

still happens with latest docker on win10

[iribeirolgc]

+1