Docker Issues and Tips (aufs/overlay/btrfs..)

Picked up and categorized subjectively from https://github.com/docker/docker/issues. Comments and pull requests are welcome.

⬜ = Open (maybe not up-to-date, please check the link by yourself!)

🔳 = Mostly resolved (ditto, plus subjective)

✅ = Resolved

Storage Drivers

AUFS

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #783	Cannot access to a directory due to a permission error	😐 Medium	😃 Easy	Expected AUFS behavior. dirperm1 mount option fixes this issue.	Update the kernel (AUFS >= 2008xxxx?) and Docker daemon (>= 1.7)	Confirm: `docker info
✅ #18180	A process becomes a zombie and hangs up	😱 High	😱 Hard(multiprocessor) 😃 Easy(uniprocessor)	Compatibility between the kernel and AUFS	Update the kernel (AUFS >= 20160111)	Java apps and MongoDB are known to be affected
✅ #20199	fcntl(F_SETFL, O_APPEND) is ignored and hence data can be corrupted	😱 High	😃 Easy	AUFS bug	Update the kernel (AUFS >= 20160301)	Dovecot is known to be affected
✅ #20240	Weird permission even though dirperm1 is enabled	😐 Medium	😱 Hard	AUFS bug	Update the kernel (AUFS >= 20160905)
⬜ AUFS ML 2016-03-08	Hang up related to O_DIRECT	😱 High	😃 Easy	Unanalyzed	None	Percona is known to be affected
⬜ #24309	Unable to remove files previously committed	😱 High	😃 Easy	Unanalyzed	This article seems related, but perhaps slightly different(Japanese)
🔳 #34361	AUFS + XFS hangs up	😱 High	😃 Easy	AUFS bug	Update AUFS

Non-bug issues:

• AUFS is not available in the mainline kernel．Only a few distros (Ubuntu, Boot2Docker, ..) support AUFS, but even for Ubuntu, Canonical says "AUFS will disappear".
• No support for extended attributes ("xattrs"), and might not ever get support (#1070, #8460).
• rename(2) is not fully supported ( see also #aufs--overlay-common )

Overlay

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #10180	RPMDB corruption	😱 High	😐 Medium	Expected overlay behavior	Use yum-{utils,plugins-ovl}-1.1.31-33.el7 (included in RHEL 7.2) or later. Kernel patch is also available.	Linux 4.6 or later prints human-friendly dmesg
✅ #12080	Cannot use UNIX domain sockets	😐 Medium	😃 Easy	Overlay Bug	Use Linux 4.7-rc4 or later
✅ #12327	pip fails	😱 High	😃 Easy	Overlay Bug	Use Linux 4.5 or later
✅ #19082	Weird behavior after removing the current directory	😃 Low	😃 Easy	Overlay Bug	Use Linux 4.5 or later
🔳 #19647, coreos/bugs#1095	Untar fails intermittently	😱 High	😱 Hard	Overlay Bug	Use Linux 4.13 with OVERLAY_FS_INDEX=y	Analysis is in progress in coreos/bugs#1095
⬜ #20640	Container cannot be started	😐 Medium	😱 Hard	Unanalyzed	None	Possibly identical to #16902
✅ #20950	/dev/console: operation not permitted	😱 High	😃 Easy	Kernel Bug	Use recent Linux kernels
✅ #21555	docker build fails intermittently (overlay1)	😱 High	😱 Hard	DiffDriver bug	Use Docker 1.13 or later	Overlay2 doesn't have this issue by design
✅ #24913	permissions broken after chown	😐 Medium	😃 Easy	Overlay Bug	Use Linux 4.6 or later	The overlay2 issue #28391 is due to the identical bug
✅ #25244	opaque flag not reset after directory copy up	😐 Medium	😃 Easy	Overlay Bug	Resolved in Linux 4.8 and backported to 4.4.21 and 4.7.4	npm is known to be affected
✅ machine#3327	chmod fails with EPERM	😃 Low	😃 Easy	Overlay Bug	Use Linux 4.5 or later
✅#27358	file removal weird on overlay + XFS (ftype=0)	😱 High	😃 Easy	Expected behavior	Format xfs with ftype=1
✅#34320	docker build produces weird images with CONFIG_OVERLAY_FS_REDIRECT_DIR=y	😱 High	😃 Easy	DiffDriver issue	Apply #34342 (Docker 17.08?)

Non-bug issues:

• 😱 High inode usage (resolved in overlay2, which will be available in Docker 1.12)
• Red Hat says "OverlayFS remains a Technology Preview in Red Hat Enterprise Linux 7.3 under most circumstances"
• rename(2) is not fully supported ( see also #aufs--overlay-common )
• MySQL doesn't work without touch-ing files under /var/lib/mysql: docker/for-linux#72 (comment)

AUFS / Overlay common

Non-bug issue: rename(2) is not fully supported #25409

reports about the incompatible behavior of rename(2) from the real world

Software	Report
Apache Kudu	https://issues.apache.org/jira/browse/KUDU-1419
CernVM-FS	https://sft.its.cern.ch/jira/browse/CVM-651
GPG	moby/moby#26317
NPM	npm/npm#9863
Samba	https://bugzilla.samba.org/show_bug.cgi?id=9966

BtrFS

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #19073	sendfile(2) can be unkillable	😃 Low	😃 Easy	BtrFS bug	None	Not likely to happen in production, but needs consideration for public PaaS
⬜ #20080	cgroups kmem limit leads crash and data corruption	😱 High	😃 Easy?	Btrfs bug	Avoid kmem limit configuration?

Non-bug issues:

• Slow #10161
• No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384
• Docker says BtrFS is Experimental. Red Hat says BtrFS is Tech Preview.

ZFS

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #20153	Some operations fail due to EBUSY	😐 Medium	😐 Medium	Daemon bug	Update Docker daemon

Non-bug issues:

• Docker says ZFS is not recommended for production.

DeviceMapper

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #4036	Mount fails	😱 High	😃 Easy	udev sync disabled	Use a Docker daemon binary which supports udev sync	Confirm: `docker info
⬜ #20401	Infinite “mount/remount” loop, which makes the system unresponsive	😱 High	😱 High	Unanalyzed (perhaps related to XFS)	None

Non-bug issues:

• Slow #10161
• No page sharing (e.g. same DLLs are loaded redundantly) http://comments.gmane.org/gmane.comp.sysutils.docker.devel/1384

Storage driver test tool

• dmcgowan/dsdbench: Docker Storage Driver Benchmarks and Tests

So which storage driver should I use?

It totally depends on your workload, but Docker, Inc. says AUFS and Devicemapper (direct-lvm) are "production-ready".

https://github.com/docker/docker/blob/master/docs/userguide/storagedriver/selectadriver.md#future-proofing

Although not listed in the above table, VFS driver is also attractive for its robustness.

Links:

• https://jpetazzo.github.io/assets/2015-03-03-not-so-deep-dive-into-docker-storage-drivers.html#1
• http://www.projectatomic.io/docs/filesystems/
• https://blog.jessfraz.com/post/the-brutally-honest-guide-to-docker-graphdrivers/

Anyway...

You know, containers should be "immutable" and "disposable".

For persistent data and some special temporary data, you should better consider using an external volume (docker run -v).

Links:

• https://github.com/emccode/rexray

Network

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
🔳 #5618	hang up with unregister_netdevice: waiting for lo to become free	😱 High	😱 Hard	Kernel bug	Use Linux 4.8 or later	The patch will be backported to old kernels in major distros
✅ #18776	TCP checksums are ignored	😱 High	😱 Hard	Kernel bug	Use Linux 4.4 or later	blog

Logging

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #19209	GELF driver saturates CPU	😱 High	😃 Easy	Compression	Disable compression
✅ #18057,#20600	cat /dev/zero leads to out of memory	😱 High	😃 Easy	logger's stdio handling issue	Use Docker 1.13 or later (or just disable the logging)	Related: #21181
⬜ #22497	container cannot be stopped if many logs are being printed	😱 High	😱 Hard	logger's stdio handling issue
✅ #22502	logging blocks the container	😱 High	😃 Easy	logger's stdio handling issue	Use Docker 1.11 or later	affected versions: 1.10.0

Others

Issue	Abstract	Impact	Reproducibility	Cause	Solution	Notes
✅ #17720	Docker daemon 1.9 serious performance issue	😱 High	😱 Hard	?	Use Docker 1.10
⬜ #19758	soft lockup related to show_mountinfo(), after frequent docker run	😱 High	😱 Hard	Unanalyzed (Kernel bug related to the number of processors?)	None
✅ #20670	/dev/pts unmounted on the HOST when you are using -v /dev:/dev (After that you can no longer open SSH nor xterm)	😱 High	😃 Easy	daemon bug related to mount namespace	Use Docker 1.11.1. (Or Spawn the docker daemon from systemd. Or do not use -v /dev:/dev)
✅ #20836	Daemon hangs up after frequent docker run	😱 High	😱 Hard	Daemon bug	Use Docker 1.11.1
✅ #28936	Strange permission issues with named containers on 1.12.3	😱 High	😃 Easy	Daemon bug related to SELinux)	Use Docker 1.12.4
✅ Ubuntu linux-azure #1719045	fatal error: unaligned sysUnused on Azure	😱 High	?	Ubuntu linux-azure kernel bug	Use linux-azure 4.11.0-1013.13 or later

Non-bug issues:

• docker ps is sometimes slow due to lock: #19328 (Mitigated in Docker 17.07, #31273
• EBUSY on docker rm in Linux < 3.19: #26510