Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

"docker stats" on Linux uses process virtual memory size instead of reporting actually used memory #34073

Closed
sandersaares opened this issue Jul 12, 2017 · 1 comment

Comments

@sandersaares
Copy link

commented Jul 12, 2017

Description

On Ubuntu Server 16.04, docker stats reports how much virtual memory a process is using, which is misleading if you are using managed runtimes like .NET/Java that allocate large chunks of virtual memory without actually using it.

Detailed explanation: https://stackoverflow.com/questions/561245/virtual-memory-usage-from-java-under-linux-too-much-memory-used/561450#561450

Steps to reproduce the issue:

  1. Run a container that executes an app using a managed runtime that performs its own memory management (e.g. java -Xms1024m -Xmx4096m com.example.Hello example from the above Stack Overflow post)
  2. docker stats

Describe the results you received:

docker stats reports very high memory usage, when actually this is just allocated virtual address space and not actually used memory.

Describe the results you expected:

docker stats reports actually used memory. I am given to understand that the best equivalent for the Windows "working set" metric is the sum of SHR and RES on Linux. This is what I expect to see.

Additional information you deem important (e.g. issue happens only occasionally):

Here is a pmap of a Mono executable that allocates around 700 MB of virtual address space but only uses around 75 MB of memory. Docker reports 700 MB as the usage, instead of the expected 75 MB.

https://gist.github.com/sandersaares/e022a24e61ec02eac6930773d81238be

Here is top output:

top - 10:36:50 up 57 days, 23:23,  0 users,  load average: 0.03, 0.06, 0.08
Tasks:   6 total,   1 running,   5 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.6 us,  2.7 sy,  0.8 ni, 94.1 id,  0.6 wa,  0.0 hi,  0.3 si,  0.0 st
KiB Mem :  3521728 total,   196196 free,   780852 used,  2544680 buff/cache
KiB Swap:        0 total,        0 free,        0 used.  2323020 avail Mem

  PID USER      PR  NI    VIRT    RES    SHR S %CPU %MEM     TIME+ COMMAND
    1 root      20   0    4508    560    484 S  0.0  0.0   0:00.01 sh
    5 root      20   0   18036   1548   1328 S  0.0  0.0   0:00.00 Entrypoint.sh
   20 root      20   0   18028   1540   1324 S  0.0  0.0   0:00.00 ExecuteApp.sh
   21 root      20   0  681272  69160   5000 S  0.0  2.0 716:21.97 mono
 3778 root      20   0   18240   3228   2756 S  0.0  0.1   0:00.03 bash
 3786 root      20   0   36768   3224   2616 R  0.0  0.1   0:00.00 top

Output of docker version:

Client:
 Version:      17.05.0-ce
 API version:  1.29
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:20:50 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.05.0-ce
 API version:  1.29 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   89658be
 Built:        Thu May  4 22:20:50 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 8
 Running: 8
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 17.05.0-ce
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 71
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-75-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 3.359GiB
Name: host-linux-02
ID: 5ZBQ:GJWC:PWH6:W7MM:O3B3:JONJ:DFQ2:OYC4:VEQW:67AC:RQWE:3IET
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

WARNING: No swap limit support

Additional environment details (AWS, VirtualBox, physical, etc.):

Azure VM.

@thaJeztah

This comment has been minimized.

Copy link
Member

commented Jul 13, 2017

Thanks for reporting; this looks like a duplicate of #10824 (comment), which was fixed through #32777 / docker/cli#80 in Docker 17.06

Closing this issue, because this should be fixed in 17.06, but let me know if you're still able to reproduce with that version.

Also feel free to continue the conversation, just closing for housekeeping 😅

@thaJeztah thaJeztah closed this Jul 13, 2017
@thaJeztah thaJeztah added this to the 17.06.0 milestone Jul 13, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants
You can’t perform that action at this time.