docker stuck when run out of host physical memory

[ibigquant]

Description

We have hundreds of containers (one per user) running on host. There's memory limit for each user/container (16G). Most of the time, most of users are not active and it's expected to be use swap memory if the physical memory runs out.

But, when it runs run out of host physical memory, the docker cannot be stopped or removed correctly.

$ sudo docker stop containerid
containerid
$ sudo docker rm containerid
Error response from daemon: You cannot remove a running container 95aa2645ceca094d55a13f57acf7c305e16e59ff5998c61ca5214c3e35f19007. Stop the container before attempting removal or use -f

(docker kill works OK)

Steps to reproduce the issue:

1. start docker containers and allocate some memory until host physical memory runs out

Describe the results you received:

Cannot stop container.

Describe the results you expected:

Container can be stopped correctly. And docker containers could use the swap.

Additional information you deem important (e.g. issue happens only occasionally):

Output of docker version:

Client:
 Version:      17.03.1-ce
 API version:  1.27
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64

Server:
 Version:      17.03.1-ce
 API version:  1.27 (minimum version 1.12)
 Go version:   go1.7.5
 Git commit:   c6d412e
 Built:        Mon Mar 27 17:14:09 2017
 OS/Arch:      linux/amd64
 Experimental: false

Output of docker info:

Containers: 65
 Running: 65
 Paused: 0
 Stopped: 0
Images: 41
Server Version: 17.03.1-ce
Storage Driver: aufs
 Root Dir: /bqmnt/data/var_lib_docker/aufs
 Backing Filesystem: extfs
 Dirs: 256
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 4ab9917febca54791c5f071a9d1f404867857fcc
runc version: 54296cf40ad8143b62dbcaa1d90e520a2136ddfe
init version: 949e6fa
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-79-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 125.9 GiB
Name: iZ8vbdzsluhqdi5qtw9273Z
ID: GMGM:D4Z4:RZFN:JEZJ:G2WQ:UWAF:D4WU:LILO:54WE:XNWE:4MJN:6473
Docker Root Dir: /bqmnt/data/var_lib_docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):

[thaJeztah]

thanks for reporting, this looks to be a duplicate of #33192, and possibly #33820. Also refer to #34377 (comment) which has some additional information

The TL;DR is that when the host runs out of resources, and the kernel starts to OOM kill processes, strange things can happen (e.g., the process monitoring if the container is still running is killed, in which case the docker has an "incorrect" view of the container's state). More information about limiting container resources can be found here; Limit a container's resources

And docker containers could use the swap.

Swapping is generally disabled on cloud providers (and often discouraged for production use, but YMMV).

In addition, if you want to control swapping for containers, swap accounting has to be enabled in your kernel configuration, which in your example looks to not be the case (from the output of docker info:);

WARNING: No swap limit support

See this section in the documentation on how to enable this

You can also check your system's configuration using the check-config.sh script, found in this repository

I'm closing this issue because this is a duplicate, but feel free to comment after I closed

[firelyu]

According to docker resource constraints, I wonder --kernel-memory can solve the memory exceed issue. If I set --kernel-memory for each container, and the total kernel memory is less than the host memory. Can I make sure the container is not killed by OOM?

[thaJeztah]

Just setting --memory should be ok; if you want to "overcommit" your host (i.e. allow the total amount of memory limited for containers to be more than the amount of physical memory), you can still decide to enable swapping on your host, but be aware that performance will drop significantly if this happens.

For kernel memory (don't think it's supported on docker 17.03 yet); also read https://www.kernel.org/doc/Documentation/cgroup-v1/memory.txt