🐳 Docker Issues and Tips (aufs/overlay/btrfs..)
Switch branches/tags
Nothing to show
Clone or download

README.md

Docker Issues and Tips (aufs/overlay/btrfs..)

Picked up and categorized subjectively from https://github.com/docker/docker/issues. Comments and pull requests are welcome.

⬜️ = Open (maybe not up-to-date, please check the link by yourself!)

πŸ”³ = Mostly resolved (ditto, plus subjective)

βœ… = Resolved

Storage Drivers

AUFS

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #783 Cannot access to a directory due to a permission error 😐 Medium πŸ˜ƒ Easy Expected AUFS behavior. dirperm1 mount option fixes this issue. Update the kernel (AUFS >= 2008xxxx?) and Docker daemon (>= 1.7) Confirm: `docker info
βœ… #18180 A process becomes a zombie and hangs up 😱 High 😱 Hard(multiprocessor)
πŸ˜ƒ Easy(uniprocessor)
Compatibility between the kernel and AUFS Update the kernel (AUFS >= 20160111) Java apps and MongoDB are known to be affected
βœ… #20199 fcntl(F_SETFL, O_APPEND) is ignored and hence data can be corrupted 😱 High πŸ˜ƒ Easy AUFS bug Update the kernel (AUFS >= 20160301) Dovecot is known to be affected
βœ… #20240 Weird permission even though dirperm1 is enabled 😐 Medium 😱 Hard AUFS bug Update the kernel (AUFS >= 20160905)
⬜️ AUFS ML 2016-03-08 Hang up related to O_DIRECT 😱 High πŸ˜ƒ Easy Unanalyzed None Percona is known to be affected
⬜️ #24309 Unable to remove files previously committed 😱 High πŸ˜ƒ Easy Unanalyzed This article seems related, but perhaps slightly different(Japanese)
πŸ”³ #34361 AUFS + XFS hangs up 😱 High πŸ˜ƒ Easy AUFS bug Update AUFS

Non-bug issues:

Overlay

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #10180 RPMDB corruption 😱 High 😐 Medium Expected overlay behavior Use yum-{utils,plugins-ovl}-1.1.31-33.el7 (included in RHEL 7.2) or later. Kernel patch is also available. Linux 4.6 or later prints human-friendly dmesg
βœ… #12080 Cannot use UNIX domain sockets 😐 Medium πŸ˜ƒ Easy Overlay Bug Use Linux 4.7-rc4 or later
βœ… #12327 pip fails 😱 High πŸ˜ƒ Easy Overlay Bug Use Linux 4.5 or later
βœ… #19082 Weird behavior after removing the current directory πŸ˜ƒ Low πŸ˜ƒ Easy Overlay Bug Use Linux 4.5 or later
πŸ”³ #19647, coreos/bugs#1095 Untar fails intermittently 😱 High 😱 Hard Overlay Bug Use Linux 4.13 with OVERLAY_FS_INDEX=y Analysis is in progress in coreos/bugs#1095
⬜️ #20640 Container cannot be started 😐 Medium 😱 Hard Unanalyzed None Possibly identical to #16902
βœ… #20950 /dev/console: operation not permitted 😱 High πŸ˜ƒ Easy Kernel Bug Use recent Linux kernels
βœ… #21555 docker build fails intermittently (overlay1) 😱 High 😱 Hard DiffDriver bug Use Docker 1.13 or later Overlay2 doesn't have this issue by design
βœ… #24913 permissions broken after chown 😐 Medium πŸ˜ƒ Easy Overlay Bug Use Linux 4.6 or later The overlay2 issue #28391 is due to the identical bug
βœ… #25244 opaque flag not reset after directory copy up 😐 Medium πŸ˜ƒ Easy Overlay Bug Resolved in Linux 4.8 and backported to 4.4.21 and 4.7.4 npm is known to be affected
βœ… machine#3327 chmod fails with EPERM πŸ˜ƒ Low πŸ˜ƒ Easy Overlay Bug Use Linux 4.5 or later
βœ…#27358 file removal weird on overlay + XFS (ftype=0) 😱 High πŸ˜ƒ Easy Expected behavior Format xfs with ftype=1
βœ…#34320 docker build produces weird images with CONFIG_OVERLAY_FS_REDIRECT_DIR=y 😱 High πŸ˜ƒ Easy DiffDriver issue Apply #34342 (Docker 17.08?)

Non-bug issues:

AUFS / Overlay common

Non-bug issue: rename(2) is not fully supported #25409

reports about the incompatible behavior of rename(2) from the real world

Software Report
Apache Kudu https://issues.apache.org/jira/browse/KUDU-1419
CernVM-FS https://sft.its.cern.ch/jira/browse/CVM-651
GPG https://github.com/docker/docker/issues/26317
NPM https://github.com/npm/npm/issues/9863
Samba https://bugzilla.samba.org/show_bug.cgi?id=9966

BtrFS

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #19073 sendfile(2) can be unkillable πŸ˜ƒ Low πŸ˜ƒ Easy BtrFS bug None Not likely to happen in production, but needs consideration for public PaaS
⬜️ #20080 cgroups kmem limit leads crash and data corruption 😱 High πŸ˜ƒ Easy? Btrfs bug Avoid kmem limit configuration?

Non-bug issues:

ZFS

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #20153 Some operations fail due to EBUSY 😐 Medium 😐 Medium Daemon bug Update Docker daemon

Non-bug issues:

DeviceMapper

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #4036 Mount fails 😱 High πŸ˜ƒ Easy udev sync disabled Use a Docker daemon binary which supports udev sync Confirm: `docker info
⬜️ #20401 Infinite β€œmount/remount” loop, which makes the system unresponsive 😱 High 😱 High Unanalyzed (perhaps related to XFS) None

Non-bug issues:

Storage driver test tool

So which storage driver should I use?

It totally depends on your workload, but Docker, Inc. says AUFS and Devicemapper (direct-lvm) are "production-ready".

https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/selectadriver.md#future-proofing

driver-pros-and-cons.png

Although not listed in the above table, VFS driver is also attractive for its robustness.

Links:

Anyway...

You know, containers should be "immutable" and "disposable".

For persistent data and some special temporary data, you should better consider using an external volume (docker run -v).

Links:

Network

Issue Abstract Impact Reproducibility Cause Solution Notes
πŸ”³ #5618 hang up with unregister_netdevice: waiting for lo to become free 😱 High 😱 Hard Kernel bug Use Linux 4.8 or later The patch will be backported to old kernels in major distros
βœ… #18776 TCP checksums are ignored 😱 High 😱 Hard Kernel bug Use Linux 4.4 or later blog

Logging

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #19209 GELF driver saturates CPU 😱 High πŸ˜ƒ Easy Compression Disable compression
βœ… #18057,#20600 cat /dev/zero leads to out of memory 😱 High πŸ˜ƒ Easy logger's stdio handling issue Use Docker 1.13 or later (or just disable the logging) Related: #21181
⬜️ #22497 container cannot be stopped if many logs are being printed 😱 High 😱 Hard logger's stdio handling issue
βœ… #22502 logging blocks the container 😱 High πŸ˜ƒ Easy logger's stdio handling issue Use Docker 1.11 or later affected versions: 1.10.0

Others

Issue Abstract Impact Reproducibility Cause Solution Notes
βœ… #17720 Docker daemon 1.9 serious performance issue 😱 High 😱 Hard ? Use Docker 1.10
⬜️ #19758 soft lockup related to show_mountinfo(), after frequent docker run 😱 High 😱 Hard Unanalyzed (Kernel bug related to the number of processors?) None
βœ… #20670 /dev/pts unmounted on the HOST when you are using -v /dev:/dev (After that you can no longer open SSH nor xterm) 😱 High πŸ˜ƒ Easy daemon bug related to mount namespace Use Docker 1.11.1. (Or Spawn the docker daemon from systemd. Or do not use -v /dev:/dev)
βœ… #20836 Daemon hangs up after frequent docker run 😱 High 😱 Hard Daemon bug Use Docker 1.11.1
βœ… #28936 Strange permission issues with named containers on 1.12.3 😱 High πŸ˜ƒ Easy Daemon bug related to SELinux) Use Docker 1.12.4
βœ… Ubuntu linux-azure #1719045 fatal error: unaligned sysUnused on Azure 😱 High ? Ubuntu linux-azure kernel bug Use linux-azure 4.11.0-1013.13 or later

Non-bug issues:

  • docker ps is sometimes slow due to lock: #19328 (Mitigated in Docker 17.07, #31273
  • EBUSY on docker rm in Linux < 3.19: #26510