Dockerd memory leak surrounding OOM killed containers

[josh-ferrell-sas]

Description

At some point in time a container begins being OOM killed which leads to a ramp up in memory usage by dockerd that correlates to an increase in goroutines. It's believed that the specific containers being OOM killed typically have double forked processes(uwsgi, gunicorn, etc..) The memory isn't released until docker is restarted.

Steps to reproduce the issue:

1. I haven't been able to deliberately reproduce it however I've experienced it across multiple clusters.

Describe the results you received:
Dockerd retains memory allocated during calls even after the OOM events stop.

Describe the results you expected:
Docker does not use as much memory.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:
Server: Docker Engine - Community
Engine:
Version: 19.03.4
API version: 1.40 (minimum version 1.12)
Go version: go1.12.10
Git commit: 9013bf583a
Built: Fri Oct 18 15:50:54 2019
OS/Arch: linux/amd64
Experimental: true
containerd:
Version: 1.2.10
GitCommit: b34a5c8af56e510852c35414db4c1f4fa6172339
runc:
Version: 1.0.0-rc8+dev
GitCommit: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
docker-init:
Version: 0.18.0
GitCommit: fec3683

Output of docker info:

Server:
Containers: 97
Running: 76
Paused: 0
Stopped: 21
Images: 299
Server Version: 19.03.4
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 4.20.7-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 125.9GiB
Name:
ID: 532A:75YV:HQCR:KCCR:DOOD:YWJV:S6GZ:HONI:R6XX:VWWW:CLJZ:BMEV
Docker Root Dir: /var/lib/docker
Debug Mode: true
File Descriptors: 439
Goroutines: 357
System Time: 2019-12-20T20:34:57.058852473Z
EventsListeners: 1
Registry: https://index.docker.io/v1/
Labels:
Experimental: true
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: true

Additional environment details (AWS, VirtualBox, physical, etc.): It has been observed on baremetal and openstack instances.

pprof.dockerd.alloc_objects.alloc_space.inuse_objects.inuse_space.014.pb.gz

pprof.dockerd.alloc_objects.alloc_space.inuse_objects.inuse_space.015.pb.gz

[thaJeztah]

/cc @cpuguy83 @kolyshkin

[josh-ferrell-sas]

Any updates on this?

[cpuguy83]

I tried to reproduce this some time ago and could not.

[cpuguy83]

Perhaps containers are being OOM killed because of a memory leak somewhere else (I've seen some issues regarding fluentd's async logging) in docker.
The pprof dumps you have unfortunately don't contain much.

Maybe you can grab the svg version?

[yu2003w]

I also observed similar issues. However, I'm not sure whether it's caused by docker it self or other containers.
Which tests should I run to identify the problem?

[pacoxu]

link #40856