docker storage drivers devicemapper and btrfs very slow #10161

Closed
edwintorok opened this Issue Jan 17, 2015 · 44 comments

Comments

Projects
None yet
@edwintorok

Description of problem:

Docker with devicemapper/btrfs is order of magnitude slower than with AUFS.

docker version:

Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): d344625
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): d344625

docker -D info:

Containers: 0
Images: 19
Storage Driver: devicemapper
 Pool Name: docker-8:18-402882181-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 465.4 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 1.204 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
Debug mode (server): false
Debug mode (client): true
Fds: 10
Goroutines: 11
EventsListeners: 0
Init SHA1: 623458afefaf342877bebd84ba389315167962ca
Init Path: /usr/lib/docker.io/dockerinit
WARNING: No memory limit support
WARNING: No swap limit support

uname -a:

Linux debian 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt2-1 (2014-12-08) x86_64 GNU/Linux

Environment details:

Physical, Debian GNU/Linux 8.0 (jessie).
SSD: OCZ-VERTEX2, HDD: WDC WD7501AALS-00J7B0.

How reproducible:

Always

Steps to Reproduce:

$ cat >Dockerfile <<EOF
FROM debian:jessie
ENV A 1
ENV B 2
ENV C 3
ENV D 4
ENV E 5
ENV F 6
ENV G 7
ENV H 8
EOF
  1. sudo docker info | grep Storage
  2. run once to download base image: sudo docker build -t test --no-cache .
  3. run again to measure just the ENVs: sudo time docker build -t test --no-cache .

Actual Results: Noticeable delay

Storage Driver: devicemapper
[...]
0.02user 0.02system 0:21.57elapsed 0%CPU (0avgtext+0avgdata 14024maxresident)k
0inputs+0outputs (0major+1119minor)pagefaults 0swaps

Expected Results: Almost instant, around 1 second:

Storage Driver: aufs
[...]
0.00user 0.04system 0:00.89elapsed 5%CPU (0avgtext+0avgdata 13496maxresident)k
0inputs+0outputs (0major+993minor)pagefaults 0swaps

Additional info:

Device Storage driver options time
SSD aufs <1s
HDD aufs 3s
SSD btrfs -s btrfs 8s
SSD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard 14s
HDD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard --storage-opt dm.datadev=/dev/vg0/data --storage-opt dm.metadatadev=/dev/vg0/metadata 15s
SSD devicemapper -s devicemapper 22s

I tried upgrading my kernel from 3.16.7 to 3.18.0 and a fresh Docker install became orders of magnitude slower. Turns out that in Debian's 3.18.0 kernel AUFS is not enabled, and it uses devicemapper backend, whereas in 3.16.7 there is AUFS.
This might also explain why docker seemed so slow on the production servers (running CentOS7), and used to be reasonably fast on my desktop (Debian Jessie).

With 3.16.7-ckt2-1 and aufs it takes under 1 second to run the testcase,
but with devicemapper it takes almost half a minute.
Even running aufs on a HDD is faster than running devicemapper/btrfs on an
SSD!

Everytime when switching between storage drivers I wiped /var/lib/docker
(sometimes also had to manually unmount, dmsetup clear, dmsetup remove when
containers got stuck):

$ sudo service docker stop
$ sudo rm -rf /var/lib/docker
$ sudo mkdir /var/lib/docker
... set DOCKEROPTS="-s devicemapper"` in /etc/default/docker
$ sudo service docker start

Now /var/lib/docker is on an SSD with XFS, and I noticed that docker by default uses discard which is well known to be slow.

With discard disabled on 3.16.7 it is still slow:

0.00user 0.03system 0:13.72elapsed 0%CPU (0avgtext+0avgdata 13492maxresident)k
0inputs+0outputs (0major+1015minor)pagefaults 0swaps

Also tried btrfs storage backend, or using a direct lvm on a HDD instead of a SSD to make sure that discard really has no impact, with similar results.
Direct LVM on the HDD with 3.18.0:

0.06user 0.07system 0:14.54elapsed 0%CPU (0avgtext+0avgdata 14004maxresident)k
0inputs+0outputs (0major+1145minor)pagefaults 0swaps
$ sudo docker info
Containers: 16
Images: 19
Storage Driver: devicemapper
 Pool Name: docker-8:18-269820673-pool
 Pool Blocksize: 65.54 kB
 Data file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata file: /var/lib/docker/devicemapper/devicemapper/metadata
 Data Space Used: 480.4 MB
 Data Space Total: 107.4 GB
 Metadata Space Used: 1.794 MB
 Metadata Space Total: 2.147 GB
 Library Version: 1.02.90 (2014-09-01)
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
WARNING: No memory limit support
WARNING: No swap limit support
$ sudo docker version
Client version: 1.3.3
Client API version: 1.15
Go version (client): go1.3.3
Git commit (client): d344625
OS/Arch (client): linux/amd64
Server version: 1.3.3
Server API version: 1.15
Go version (server): go1.3.3
Git commit (server): d344625

btrfs-on-SSD:

0.02user 0.04system 0:07.34elapsed 0%CPU (0avgtext+0avgdata 13836maxresident)k
0inputs+0outputs (0major+1095minor)pagefaults 0swaps

aufs-on-HDD:

0.01user 0.04system 0:02.39elapsed 2%CPU (0avgtext+0avgdata 13936maxresident)k
0inputs+0outputs (0major+1109minor)pagefaults 0swaps

Edit: added btrs-on-SSD numbers, aufs-on-HDD numbers and checked all timings again.

Apparently AUFS is the only storage backend that is usable in a production environment with adequate performance, except that is the only place where it is not available ...

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Jan 18, 2015

Contributor

This is likely because aufs uses aufs itself to get/apply diffs, where as btrfs and devmapper use the default differ.

Also, would be worthwhile to test overlay on the 3.18 kernel.

Contributor

cpuguy83 commented Jan 18, 2015

This is likely because aufs uses aufs itself to get/apply diffs, where as btrfs and devmapper use the default differ.

Also, would be worthwhile to test overlay on the 3.18 kernel.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 Jan 18, 2015

Contributor

I think this is because of bug, which was fixed in #9720
@edwintorok Could you retest with master?

Contributor

LK4D4 commented Jan 18, 2015

I think this is because of bug, which was fixed in #9720
@edwintorok Could you retest with master?

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Jan 18, 2015

@cpuguy83 the docker version in Debian is too old for overlayfs, but I downloaded a newer one and started it with -s overlay:

$ docker info
Containers: 0
Images: 27
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 11.66 GiB
Name: debian
ID: ECOC:PJPR:SFLR:PTOJ:QZJO:FDUE:PYJM:OIC4:DVN6:TSSC:PLMJ:5BNI
WARNING: No memory limit support
WARNING: No swap limit support

$ docker version
[..]
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

Timing is about same as btrfs (better than devicemapper, worse than aufs)

0.01user 0.01system 0:08.38elapsed 0%CPU (0avgtext+0avgdata 12016maxresident)k
0inputs+0outputs (0major+822minor)pagefaults 0swaps

Which differ does overlayfs use?

@cpuguy83 the docker version in Debian is too old for overlayfs, but I downloaded a newer one and started it with -s overlay:

$ docker info
Containers: 0
Images: 27
Storage Driver: overlay
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 11.66 GiB
Name: debian
ID: ECOC:PJPR:SFLR:PTOJ:QZJO:FDUE:PYJM:OIC4:DVN6:TSSC:PLMJ:5BNI
WARNING: No memory limit support
WARNING: No swap limit support

$ docker version
[..]
Server version: 1.4.1
Server API version: 1.16
Go version (server): go1.3.3
Git commit (server): 5bc2ff8

Timing is about same as btrfs (better than devicemapper, worse than aufs)

0.01user 0.01system 0:08.38elapsed 0%CPU (0avgtext+0avgdata 12016maxresident)k
0inputs+0outputs (0major+822minor)pagefaults 0swaps

Which differ does overlayfs use?

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Jan 18, 2015

@LK4D4 tested devicemapper on master, it is a bit faster than before, but still quite slow compared to aufs:

$ docker version
Client version: 1.4.1-dev
Client API version: 1.16
Go version (client): go1.4.1
Git commit (client): 467b7c8
OS/Arch (client): linux/amd64
Server version: 1.4.1-dev
Server API version: 1.16
Go version (server): go1.4.1
Git commit (server): 467b7c8

$ docker info
Containers: 1
Images: 58
Storage Driver: devicemapper
 Pool Name: docker-8:34-2066085-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: <unknown>
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.554 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 3.879 MB
 Metadata Space Total: 2.147 GB
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 11.66 GiB
Name: debian
ID: ECOC:PJPR:SFLR:PTOJ:QZJO:FDUE:PYJM:OIC4:DVN6:TSSC:PLMJ:5BNI
WARNING: No memory limit support
WARNING: No swap limit support

With defaults:

0.01user 0.01system 0:18.62elapsed 0%CPU (0avgtext+0avgdata 12704maxresident)k
0inputs+0outputs (0major+842minor)pagefaults 0swaps

With discards disabled:

0.00user 0.01system 0:09.49elapsed 0%CPU (0avgtext+0avgdata 12660maxresident)k
0inputs+0outputs (0major+845minor)pagefaults 0swaps

@LK4D4 tested devicemapper on master, it is a bit faster than before, but still quite slow compared to aufs:

$ docker version
Client version: 1.4.1-dev
Client API version: 1.16
Go version (client): go1.4.1
Git commit (client): 467b7c8
OS/Arch (client): linux/amd64
Server version: 1.4.1-dev
Server API version: 1.16
Go version (server): go1.4.1
Git commit (server): 467b7c8

$ docker info
Containers: 1
Images: 58
Storage Driver: devicemapper
 Pool Name: docker-8:34-2066085-pool
 Pool Blocksize: 65.54 kB
 Backing Filesystem: <unknown>
 Data file: /dev/loop0
 Metadata file: /dev/loop1
 Data Space Used: 2.554 GB
 Data Space Total: 107.4 GB
 Metadata Space Used: 3.879 MB
 Metadata Space Total: 2.147 GB
 Data loop file: /var/lib/docker/devicemapper/devicemapper/data
 Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
 Library Version: 1.02.82-git (2013-10-04)
Execution Driver: native-0.2
Kernel Version: 3.18.0-trunk-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 8
Total Memory: 11.66 GiB
Name: debian
ID: ECOC:PJPR:SFLR:PTOJ:QZJO:FDUE:PYJM:OIC4:DVN6:TSSC:PLMJ:5BNI
WARNING: No memory limit support
WARNING: No swap limit support

With defaults:

0.01user 0.01system 0:18.62elapsed 0%CPU (0avgtext+0avgdata 12704maxresident)k
0inputs+0outputs (0major+842minor)pagefaults 0swaps

With discards disabled:

0.00user 0.01system 0:09.49elapsed 0%CPU (0avgtext+0avgdata 12660maxresident)k
0inputs+0outputs (0major+845minor)pagefaults 0swaps
@unclejack

This comment has been minimized.

Show comment
Hide comment
@unclejack

unclejack Feb 2, 2015

Contributor

@edwintorok I couldn't see in the table from #10161 (comment) the results for the SSD with non-default devicemapper options.

Have you tested that as well?

Contributor

unclejack commented Feb 2, 2015

@edwintorok I couldn't see in the table from #10161 (comment) the results for the SSD with non-default devicemapper options.

Have you tested that as well?

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Feb 2, 2015

@unclejack : I tested SSD with discard disabled, I haven't tested SSD with dm.datadev though because I don't have LVM set up on this SSD:
"SSD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard 14s"

@unclejack : I tested SSD with discard disabled, I haven't tested SSD with dm.datadev though because I don't have LVM set up on this SSD:
"SSD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard 14s"

@yonah-codefresh

This comment has been minimized.

Show comment
Hide comment
@yonah-codefresh

yonah-codefresh May 21, 2015

Is there any progress on this?
I tried running:
Ubuntu 15.04
kernel 3.19.0-18
docker 1.6.1 with overlayfs on top of ext4 formatted with -T news

Building an image took 5 minutes.
After switching back to aufs on the same machine, the same build takes 15 seconds.

Is there any progress on this?
I tried running:
Ubuntu 15.04
kernel 3.19.0-18
docker 1.6.1 with overlayfs on top of ext4 formatted with -T news

Building an image took 5 minutes.
After switching back to aufs on the same machine, the same build takes 15 seconds.

@LK4D4

This comment has been minimized.

Show comment
Hide comment
@LK4D4

LK4D4 May 21, 2015

Contributor

@yonah-codefresh Can you provide dockerfile? Maybe we have bug in some particular instruction.

Contributor

LK4D4 commented May 21, 2015

@yonah-codefresh Can you provide dockerfile? Maybe we have bug in some particular instruction.

@mbentley

This comment has been minimized.

Show comment
Hide comment
@mbentley

mbentley Jun 1, 2015

Contributor

I'm also seeing incredibly slow performance with non-aufs graph drivers. It seems to depend on the image I am using in my Dockerfile to start my build from. I have three examples here in this repo where I am using scratch, debian:jessie and then gcc:4.9 in my FROM:
https://github.com/mbentley/docker_examples/tree/master/build_speed_test

System Info

This is on an Intel i7 920 with an Intel X25-M SSD.

# uname -a
Linux blackbox 4.0.0-1-amd64 #1 SMP Debian 4.0.2-1 (2015-05-11) x86_64 GNU/Linux
# docker --version
Docker version 1.6.2, build 7c8fca2
# lsb_release -s -d
Debian GNU/Linux 8.0 (jessie)

Here are results with aufs, devicemapper, and overlay:

aufs graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m0.481s
    user    0m0.020s
    sys     0m0.000s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m0.577s
    user    0m0.016s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m0.547s
    user    0m0.012s
    sys     0m0.008s
    

devicemapper graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m5.149s
    user    0m0.012s
    sys     0m0.008s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m8.329s
    user    0m0.020s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m51.638s
    user    0m0.012s
    sys     0m0.008s
    

overlay graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m0.650s
    user    0m0.016s
    sys     0m0.004s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m3.793s
    user    0m0.020s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m48.403s
    user    0m0.016s
    sys     0m0.004s
    
Contributor

mbentley commented Jun 1, 2015

I'm also seeing incredibly slow performance with non-aufs graph drivers. It seems to depend on the image I am using in my Dockerfile to start my build from. I have three examples here in this repo where I am using scratch, debian:jessie and then gcc:4.9 in my FROM:
https://github.com/mbentley/docker_examples/tree/master/build_speed_test

System Info

This is on an Intel i7 920 with an Intel X25-M SSD.

# uname -a
Linux blackbox 4.0.0-1-amd64 #1 SMP Debian 4.0.2-1 (2015-05-11) x86_64 GNU/Linux
# docker --version
Docker version 1.6.2, build 7c8fca2
# lsb_release -s -d
Debian GNU/Linux 8.0 (jessie)

Here are results with aufs, devicemapper, and overlay:

aufs graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m0.481s
    user    0m0.020s
    sys     0m0.000s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m0.577s
    user    0m0.016s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m0.547s
    user    0m0.012s
    sys     0m0.008s
    

devicemapper graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m5.149s
    user    0m0.012s
    sys     0m0.008s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m8.329s
    user    0m0.020s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m51.638s
    user    0m0.012s
    sys     0m0.008s
    

overlay graph driver

  • scratch: time docker build --no-cache -f Dockerfile.scratch -t test .

    real    0m0.650s
    user    0m0.016s
    sys     0m0.004s
    
  • debian:jessie: time docker build --no-cache -f Dockerfile.jessie -t test .

    real    0m3.793s
    user    0m0.020s
    sys     0m0.000s
    
  • gcc:4.9: time docker build --no-cache -f Dockerfile.gcc -t test .

    real    0m48.403s
    user    0m0.016s
    sys     0m0.004s
    
@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Jul 29, 2015

FWIW I tried overlayfs on Linux 4.1.2 again, and the speed hasn't improved. Testing with some real Dockerfiles shows that it is also buggy none of Debian/CentOS/Ubuntu images can update their package cache, so the benchmark results for overlayfs above should probably be ignored:

time="2015-07-29T20:02:34+03:00" level=info msg="lstat /var/lib/docker/overlay/c5e1c19ad9d2132bffa93e93d94f8d9598648b0acd3d73a6c5ac38941191fafa/merged/var/lib/yum/rpmdb-indexes/conflicts: no such file or directory"

time="2015-07-29T20:03:25+03:00" level=info msg="lstat /var/lib/docker/overlay/708f7cd34c237a5f805b1e1e4efdb336b612774c6fbdccca67dac586a0db6581/merged/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise-updates_main_binary-amd64_Packages: no such file or directory" 

time="2015-07-29T20:03:56+03:00" level=info msg="lstat /var/lib/docker/overlay/06ec8e899dd017a524c0980f7642e1cd319a2d8223fe9f99b798cff10d263932/merged/var/cache/apt/pkgcache.bin: no such file or directory"

time="2015-07-29T20:04:46+03:00" level=info msg="lstat /var/lib/docker/overlay/4d15a94fe1bf23dda2584bf2bd14167efea89db7f417c0f96a96127b024454f0/merged/usr/lib/gcc/i586-linux-gnu/4.9.1: no such file or directory" 

FWIW I tried overlayfs on Linux 4.1.2 again, and the speed hasn't improved. Testing with some real Dockerfiles shows that it is also buggy none of Debian/CentOS/Ubuntu images can update their package cache, so the benchmark results for overlayfs above should probably be ignored:

time="2015-07-29T20:02:34+03:00" level=info msg="lstat /var/lib/docker/overlay/c5e1c19ad9d2132bffa93e93d94f8d9598648b0acd3d73a6c5ac38941191fafa/merged/var/lib/yum/rpmdb-indexes/conflicts: no such file or directory"

time="2015-07-29T20:03:25+03:00" level=info msg="lstat /var/lib/docker/overlay/708f7cd34c237a5f805b1e1e4efdb336b612774c6fbdccca67dac586a0db6581/merged/var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_precise-updates_main_binary-amd64_Packages: no such file or directory" 

time="2015-07-29T20:03:56+03:00" level=info msg="lstat /var/lib/docker/overlay/06ec8e899dd017a524c0980f7642e1cd319a2d8223fe9f99b798cff10d263932/merged/var/cache/apt/pkgcache.bin: no such file or directory"

time="2015-07-29T20:04:46+03:00" level=info msg="lstat /var/lib/docker/overlay/4d15a94fe1bf23dda2584bf2bd14167efea89db7f417c0f96a96127b024454f0/merged/usr/lib/gcc/i586-linux-gnu/4.9.1: no such file or directory" 
@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Jul 29, 2015

Member

Yes, that's a known issue with overlay, and currently one of the biggest show-stoppers; RPM and overlay don't like each other

Member

thaJeztah commented Jul 29, 2015

Yes, that's a known issue with overlay, and currently one of the biggest show-stoppers; RPM and overlay don't like each other

@hasufell

This comment has been minimized.

Show comment
Hide comment
@hasufell

hasufell Aug 27, 2015

This is quite important imo. aufs is buggy and caused random failures for me with programs that use the atomic rename syscall without falling back to some sort of "copy and rm" method. So the only sane alternative was devicemapper, but it really is slow which is quite a nuisance if you are currently developing a Dockerfile.

This is quite important imo. aufs is buggy and caused random failures for me with programs that use the atomic rename syscall without falling back to some sort of "copy and rm" method. So the only sane alternative was devicemapper, but it really is slow which is quite a nuisance if you are currently developing a Dockerfile.

@mogthesprog

This comment has been minimized.

Show comment
Hide comment
@mogthesprog

mogthesprog Sep 20, 2015

Hi Guys,

Has there been any further development on this issue?

Hi Guys,

Has there been any further development on this issue?

@GordonTheTurtle

This comment has been minimized.

Show comment
Hide comment
@GordonTheTurtle

GordonTheTurtle Sep 20, 2015

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@poldridge

USER POLL

The best way to get notified when there are changes in this discussion is by clicking the Subscribe button in the top right.

The people listed below have appreciated your meaningfull discussion with a random +1:

@poldridge

@butonic butonic referenced this issue in owncloud/core Oct 5, 2015

Merged

Swift primary storage tests #19414

@priestlyemu

This comment has been minimized.

Show comment
Hide comment
@priestlyemu

priestlyemu Nov 19, 2015

I am also seeing this. Had been using aufs, but it was buggy (deleted files in container would reappear), so I switched. Building containers slowed to a crawl. However, the bug I was seeing with aufs went away, so there's that.

My environment:
mke% lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

mke% uname -a
Linux mke 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

mke% docker info
Containers: 2
Images: 23
Server Version: 1.9.0
Storage Driver: devicemapper
Pool Name: docker-252:0-4980737-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.818 GB
Data Space Total: 107.4 GB
Data Space Available: 104.6 GB
Metadata Space Used: 2.937 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-33-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 15.67 GiB
Name: mke
ID: VZW5:L52X:FSEY:I5JR:PQNX:BJ3H:7RBO:VW4G:DESH:KM2N:IGSX:OS2H
WARNING: No swap limit support

I am also seeing this. Had been using aufs, but it was buggy (deleted files in container would reappear), so I switched. Building containers slowed to a crawl. However, the bug I was seeing with aufs went away, so there's that.

My environment:
mke% lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 14.04.3 LTS
Release: 14.04
Codename: trusty

mke% uname -a
Linux mke 3.19.0-33-generic #38~14.04.1-Ubuntu SMP Fri Nov 6 18:17:28 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

mke% docker info
Containers: 2
Images: 23
Server Version: 1.9.0
Storage Driver: devicemapper
Pool Name: docker-252:0-4980737-pool
Pool Blocksize: 65.54 kB
Base Device Size: 107.4 GB
Backing Filesystem: extfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 2.818 GB
Data Space Total: 107.4 GB
Data Space Available: 104.6 GB
Metadata Space Used: 2.937 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.145 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.77 (2012-10-15)
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.19.0-33-generic
Operating System: Ubuntu 14.04.3 LTS
CPUs: 1
Total Memory: 15.67 GiB
Name: mke
ID: VZW5:L52X:FSEY:I5JR:PQNX:BJ3H:7RBO:VW4G:DESH:KM2N:IGSX:OS2H
WARNING: No swap limit support

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 22, 2015

Member

For those here experiencing slow performance on devicemapper, make sure you're not using loopback. Read our new documentation on setting up devicemapper for production; https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

Also, the new storage driver section gives a lot more information on differences between the various drivers; https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

Member

thaJeztah commented Nov 22, 2015

For those here experiencing slow performance on devicemapper, make sure you're not using loopback. Read our new documentation on setting up devicemapper for production; https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

Also, the new storage driver section gives a lot more information on differences between the various drivers; https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Nov 22, 2015

On 11/22/2015 08:40 PM, Sebastiaan van Stijn wrote:

For those here experiencing slow performance on devicemapper, make sure you're /not/ using loopback. Read our new documentation on setting up devicemapper for production; https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

Also, the new storage driver section gives a lot more information on differences between the various drivers; https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

In the original comment I did test direct LVM mode, and it was significantly slower than with AUFS:
HDD aufs 3s
HDD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard --storage-opt dm.datadev=/dev/vg0/data --storage-opt dm.metadatadev=/dev/vg0/metadata 15s

According to #10161 (comment) the problem is the differ, which is much slower than the one used for AUFS.
Would it be possible to improve its performance?

On 11/22/2015 08:40 PM, Sebastiaan van Stijn wrote:

For those here experiencing slow performance on devicemapper, make sure you're /not/ using loopback. Read our new documentation on setting up devicemapper for production; https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/

Also, the new storage driver section gives a lot more information on differences between the various drivers; https://docs.docker.com/engine/userguide/storagedriver/imagesandcontainers/

In the original comment I did test direct LVM mode, and it was significantly slower than with AUFS:
HDD aufs 3s
HDD devicemapper -s devicemapper --storage-opt dm.blkdiscard=false --storage-opt dm.mountopt=nodiscard --storage-opt dm.datadev=/dev/vg0/data --storage-opt dm.metadatadev=/dev/vg0/metadata 15s

According to #10161 (comment) the problem is the differ, which is much slower than the one used for AUFS.
Would it be possible to improve its performance?

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Nov 27, 2015

I've been using docker in a production environment for about 1 year now and my experience is very similar. AUFS is faster but the bugs are unbearable. Deleted files showing back up and random crashes. Then switched to devicemapper but building a Dockerfile takes longer than building an entire server from scratch!

Really tough decision, dreadful performance vs inexplicably buggy ...

I've been using docker in a production environment for about 1 year now and my experience is very similar. AUFS is faster but the bugs are unbearable. Deleted files showing back up and random crashes. Then switched to devicemapper but building a Dockerfile takes longer than building an entire server from scratch!

Really tough decision, dreadful performance vs inexplicably buggy ...

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Nov 27, 2015

Contributor

If device mapper is that slow, then something is wrong with your configuration. Note that the default configuration is not suitable for prod.

Also not sure that I've seen the issues you mentioned with aufs... That sucks...

Contributor

cpuguy83 commented Nov 27, 2015

If device mapper is that slow, then something is wrong with your configuration. Note that the default configuration is not suitable for prod.

Also not sure that I've seen the issues you mentioned with aufs... That sucks...

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Nov 27, 2015

I understand there are improvements to be done in Prod but it really shouldn't be that bad. The AUFS issue is a known issue which as far as I know hasn't been fully resolved: #9690

I understand there are improvements to be done in Prod but it really shouldn't be that bad. The AUFS issue is a known issue which as far as I know hasn't been fully resolved: #9690

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 27, 2015

Member

I saw that issue was a duplicate of #9093, and see some PRs linked to that. Wondering if that has been resolved since (I just left a comment on that issue)

Member

thaJeztah commented Nov 27, 2015

I saw that issue was a duplicate of #9093, and see some PRs linked to that. Wondering if that has been resolved since (I just left a comment on that issue)

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Nov 27, 2015

Contributor

@alercunha try setting the backing store for device mapper to xfs. We're thinking about making that default.
It resolved some really bad performance issues provisioning new layers in device mapper.

Contributor

cpuguy83 commented Nov 27, 2015

@alercunha try setting the backing store for device mapper to xfs. We're thinking about making that default.
It resolved some really bad performance issues provisioning new layers in device mapper.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Nov 27, 2015

Member

Just got confirmation that #9093 should be resolved, so it may be worth trying aufs again.

Member

thaJeztah commented Nov 27, 2015

Just got confirmation that #9093 should be resolved, so it may be worth trying aufs again.

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Nov 27, 2015

@cpuguy83 yes I am reading some documentation and will test in the near future... right now I will test AUFS again since it seems to be the (unofficial) recommended driver and visibly faster than devicemapper. Once I have some results I will come back to report but for now just wanted to confirm that I have experienced similar issues reported earlier in this thread.

@cpuguy83 yes I am reading some documentation and will test in the near future... right now I will test AUFS again since it seems to be the (unofficial) recommended driver and visibly faster than devicemapper. Once I have some results I will come back to report but for now just wanted to confirm that I have experienced similar issues reported earlier in this thread.

@gourao

This comment has been minimized.

Show comment
Hide comment
@gourao

gourao Dec 4, 2015

Contributor

@edwintorok - Just for completeness, would it be possible to get the same matrix completed with ZFS as well? I don't have your same exact config for an apples to apples comparison, but I think it would be interesting to complete this matrix with ZFS. Sorry to ask!

Contributor

gourao commented Dec 4, 2015

@edwintorok - Just for completeness, would it be possible to get the same matrix completed with ZFS as well? I don't have your same exact config for an apples to apples comparison, but I think it would be interesting to complete this matrix with ZFS. Sorry to ask!

@edwintorok

This comment has been minimized.

Show comment
Hide comment
@edwintorok

edwintorok Dec 4, 2015

On 12/04/2015 05:26 AM, Gou Rao wrote:

@edwintorok https://github.com/edwintorok - Just for completeness, would it be possible to get the same matrix completed with ZFS as well? I don't have your same exact config for an apples to apples comparison, but I think it would be interesting to complete this matrix with ZFS. Sorry to ask!

I only have zfs-fuse available on Debian, is that what I should test?

On 12/04/2015 05:26 AM, Gou Rao wrote:

@edwintorok https://github.com/edwintorok - Just for completeness, would it be possible to get the same matrix completed with ZFS as well? I don't have your same exact config for an apples to apples comparison, but I think it would be interesting to complete this matrix with ZFS. Sorry to ask!

I only have zfs-fuse available on Debian, is that what I should test?

@gourao

This comment has been minimized.

Show comment
Hide comment
@gourao

gourao Dec 5, 2015

Contributor

I don't expect zfs via fuse to actually do any better so I suppose that is not a useful test.

Contributor

gourao commented Dec 5, 2015

I don't expect zfs via fuse to actually do any better so I suppose that is not a useful test.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 5, 2015

Contributor

zfs is just using the fs differ anyway, so same issue as btrfs. Initially when zfs was being implemented it was using the built-in zfs tooling, but was actually slower than the fs diff (IIRC)...

I've looked at using btrfs send to implement diffing for the btrfs driver, but it is not as trivial as it sounds.

Contributor

cpuguy83 commented Dec 5, 2015

zfs is just using the fs differ anyway, so same issue as btrfs. Initially when zfs was being implemented it was using the built-in zfs tooling, but was actually slower than the fs diff (IIRC)...

I've looked at using btrfs send to implement diffing for the btrfs driver, but it is not as trivial as it sounds.

@gourao

This comment has been minimized.

Show comment
Hide comment
@gourao

gourao Dec 8, 2015

Contributor

@cpuguy83 The problems lie in 3 areas, as far as I can see

  1. Non overlay (aufs and overlayfs) graph drivers cause problems because they pressure page cache (same file, different inodes). These are problems that plague btrfs, zfs, devicemapper etc.
  2. Overlay abuses inode count on modified files.
  3. Additionally overlay has known issues with correctness (#10180)

I am working on a prototype that can hopefuly see if the above can be mitigated. Additionally I'll see if your alternate diffing mechanism can work in this context.

Contributor

gourao commented Dec 8, 2015

@cpuguy83 The problems lie in 3 areas, as far as I can see

  1. Non overlay (aufs and overlayfs) graph drivers cause problems because they pressure page cache (same file, different inodes). These are problems that plague btrfs, zfs, devicemapper etc.
  2. Overlay abuses inode count on modified files.
  3. Additionally overlay has known issues with correctness (#10180)

I am working on a prototype that can hopefuly see if the above can be mitigated. Additionally I'll see if your alternate diffing mechanism can work in this context.

@Yajo

This comment has been minimized.

Show comment
Hide comment
@Yajo

Yajo Aug 22, 2016

So then is this a bug in devicemapper driver, or just bad configuration?

I ask because I'm using the default configuration that ships with Fedora in my development laptop (no AUFS AFAIK), and builds are awfully slow; if problem is configuration, then defaults should be better IMHO.

I hope this helps:

$ docker info
Containers: 38
 Running: 4
 Paused: 0
 Stopped: 34
Images: 77
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: fedora-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 37.76 GB
 Data Space Total: 42.9 GB
 Data Space Available: 5.144 GB
 Metadata Space Used: 12.03 MB
 Metadata Space Total: 926.9 MB
 Metadata Space Available: 914.9 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.122 (2016-04-09)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.6.6-300.fc24.x86_64
Operating System: Fedora 24 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 4
Total Memory: 6.761 GiB
Name: yajolap
ID: VEIH:O2O5:3OG6:YVMU:TPDO:JYRA:O5WO:GIOE:23OU:3S45:U7YA:VRAG
Registries: docker.io (secure)

$ cat /usr/lib/docker-storage-setup/docker-storage-setup
# Specify storage driver one wants to use with docker. Default is devicemapper.
# Other possible options are overlay and "". Empty string means do not do
# any storage setup.
#
STORAGE_DRIVER=devicemapper

# A set of extra options that should be appended to the generated
# DOCKER_STORAGE_OPTIONS string. These options will be passed to the Docker
# daemon as-is and should be valid Docker storage options.
# EXTRA_DOCKER_STORAGE_OPTIONS="--storage-opt dm.fs=ext4"

# A quoted, space-separated list of devices to be used.  This currently
# expects the devices to be unpartitioned drives.  If "VG" is not specified,
# then use of the root disk's extra space is implied.
#
# DEVS=/dev/vdb

# The volume group to use for docker storage.  Defaults to the
# volume group where the root filesystem resides.  If VG is specified and the
# volume group does not exist, it will be created (which requires that "DEVS"
# be nonempty, since we don't currently support putting a second partition on
# the root disk).
#
# VG=

# The size to which the root filesystem should be grown.
# Value should be acceptable to -L option of lvextend.
#
# ROOT_SIZE=8G

# The desired size for the docker data LV.  It defaults using 60% of all
# free space.
#
# DATA_SIZE can take values acceptable to "lvcreate -L" as well as some
# values acceptable to to "lvcreate -l". If user intends to pass values
# acceptable to "lvcreate -l", then only those values which contains "%"
# in syntax are acceptable.  If value does not contain "%" it is assumed
# value is suitable for "lvcreate -L".
#
DATA_SIZE=40%FREE

# MIN_DATA_SIZE specifies the minimum size of data volume otherwise pool
# creation fails.
#
# Value should be a number followed by a optional suffix. "bBsSkKmMgGtTpPeE"
# are valid suffixes. If no suffix is specified then value will be considered
# as mebibyte unit.
#
# Both upper and lower case suffix represent same unit of size. Use suffix B
# for Bytes, S for sectors as 512 bytes, K for kibibytes (1024 bytes), M for
# mebibytes (1024 kibibytes), G for gibibytes, T for tebibytes, P for
# pebibytes and E for exbibytes.
#
MIN_DATA_SIZE=2G

# Controls the chunk size/block size of thin pool. Value of CHUNK_SIZE
# be suitable to be passed to --chunk-size option of lvconvert.
#
CHUNK_SIZE=512K

# Enable resizing partition table backing root volume group. By default it
# is disabled until and unless GROWPART=true is specified.
#
GROWPART=false

# Enable/disable automatic pool extension using lvm
AUTO_EXTEND_POOL=yes

# Auto pool extension threshold (in % of pool size)
POOL_AUTOEXTEND_THRESHOLD=60

# Extend the pool by specified percentage when threshold is hit.
POOL_AUTOEXTEND_PERCENT=20

# Device wait timeout in seconds. This is generic timeout which can be used by
# docker storage setup service to wait on various kind of block devices.
# Setting a value of 0 can disable this wait.
DEVICE_WAIT_TIMEOUT=60

# Wipe any signatures (partition, filesystem, lvm etc) found on disk.
# This could mean wiping the signature explicitly or using force options
# of various commands to wipe/overwrite signatures. By default signatures
# are not wiped and user needs to wipe these. One can change default behavior
# by setting WIPE_SIGNATURES=true. Be careful before using this option
# as this means if there was any leftover data on disk, it will be lost.
WIPE_SIGNATURES=false

Yajo commented Aug 22, 2016

So then is this a bug in devicemapper driver, or just bad configuration?

I ask because I'm using the default configuration that ships with Fedora in my development laptop (no AUFS AFAIK), and builds are awfully slow; if problem is configuration, then defaults should be better IMHO.

I hope this helps:

$ docker info
Containers: 38
 Running: 4
 Paused: 0
 Stopped: 34
Images: 77
Server Version: 1.10.3
Storage Driver: devicemapper
 Pool Name: fedora-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 107.4 GB
 Backing Filesystem: xfs
 Data file: 
 Metadata file: 
 Data Space Used: 37.76 GB
 Data Space Total: 42.9 GB
 Data Space Available: 5.144 GB
 Metadata Space Used: 12.03 MB
 Metadata Space Total: 926.9 MB
 Metadata Space Available: 914.9 MB
 Udev Sync Supported: true
 Deferred Removal Enabled: true
 Deferred Deletion Enabled: true
 Deferred Deleted Device Count: 0
 Library Version: 1.02.122 (2016-04-09)
Execution Driver: native-0.2
Logging Driver: journald
Plugins: 
 Volume: local
 Network: bridge null host
Kernel Version: 4.6.6-300.fc24.x86_64
Operating System: Fedora 24 (Workstation Edition)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 0
CPUs: 4
Total Memory: 6.761 GiB
Name: yajolap
ID: VEIH:O2O5:3OG6:YVMU:TPDO:JYRA:O5WO:GIOE:23OU:3S45:U7YA:VRAG
Registries: docker.io (secure)

$ cat /usr/lib/docker-storage-setup/docker-storage-setup
# Specify storage driver one wants to use with docker. Default is devicemapper.
# Other possible options are overlay and "". Empty string means do not do
# any storage setup.
#
STORAGE_DRIVER=devicemapper

# A set of extra options that should be appended to the generated
# DOCKER_STORAGE_OPTIONS string. These options will be passed to the Docker
# daemon as-is and should be valid Docker storage options.
# EXTRA_DOCKER_STORAGE_OPTIONS="--storage-opt dm.fs=ext4"

# A quoted, space-separated list of devices to be used.  This currently
# expects the devices to be unpartitioned drives.  If "VG" is not specified,
# then use of the root disk's extra space is implied.
#
# DEVS=/dev/vdb

# The volume group to use for docker storage.  Defaults to the
# volume group where the root filesystem resides.  If VG is specified and the
# volume group does not exist, it will be created (which requires that "DEVS"
# be nonempty, since we don't currently support putting a second partition on
# the root disk).
#
# VG=

# The size to which the root filesystem should be grown.
# Value should be acceptable to -L option of lvextend.
#
# ROOT_SIZE=8G

# The desired size for the docker data LV.  It defaults using 60% of all
# free space.
#
# DATA_SIZE can take values acceptable to "lvcreate -L" as well as some
# values acceptable to to "lvcreate -l". If user intends to pass values
# acceptable to "lvcreate -l", then only those values which contains "%"
# in syntax are acceptable.  If value does not contain "%" it is assumed
# value is suitable for "lvcreate -L".
#
DATA_SIZE=40%FREE

# MIN_DATA_SIZE specifies the minimum size of data volume otherwise pool
# creation fails.
#
# Value should be a number followed by a optional suffix. "bBsSkKmMgGtTpPeE"
# are valid suffixes. If no suffix is specified then value will be considered
# as mebibyte unit.
#
# Both upper and lower case suffix represent same unit of size. Use suffix B
# for Bytes, S for sectors as 512 bytes, K for kibibytes (1024 bytes), M for
# mebibytes (1024 kibibytes), G for gibibytes, T for tebibytes, P for
# pebibytes and E for exbibytes.
#
MIN_DATA_SIZE=2G

# Controls the chunk size/block size of thin pool. Value of CHUNK_SIZE
# be suitable to be passed to --chunk-size option of lvconvert.
#
CHUNK_SIZE=512K

# Enable resizing partition table backing root volume group. By default it
# is disabled until and unless GROWPART=true is specified.
#
GROWPART=false

# Enable/disable automatic pool extension using lvm
AUTO_EXTEND_POOL=yes

# Auto pool extension threshold (in % of pool size)
POOL_AUTOEXTEND_THRESHOLD=60

# Extend the pool by specified percentage when threshold is hit.
POOL_AUTOEXTEND_PERCENT=20

# Device wait timeout in seconds. This is generic timeout which can be used by
# docker storage setup service to wait on various kind of block devices.
# Setting a value of 0 can disable this wait.
DEVICE_WAIT_TIMEOUT=60

# Wipe any signatures (partition, filesystem, lvm etc) found on disk.
# This could mean wiping the signature explicitly or using force options
# of various commands to wipe/overwrite signatures. By default signatures
# are not wiped and user needs to wipe these. One can change default behavior
# by setting WIPE_SIGNATURES=true. Be careful before using this option
# as this means if there was any leftover data on disk, it will be lost.
WIPE_SIGNATURES=false
@Yajo

This comment has been minimized.

Show comment
Hide comment
@Yajo

Yajo Sep 30, 2016

Has anybody found any workaround to this?

Yajo commented Sep 30, 2016

Has anybody found any workaround to this?

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Sep 30, 2016

@Yajo yes, use AUFS ;)

@Yajo yes, use AUFS ;)

@Yajo

This comment has been minimized.

Show comment
Hide comment
@Yajo

Yajo Sep 30, 2016

And if I have other thing than Ubuntu Fedora for development?

Yajo commented Sep 30, 2016

And if I have other thing than Ubuntu Fedora for development?

@priestlyemu

This comment has been minimized.

Show comment
Hide comment
@priestlyemu

priestlyemu Sep 30, 2016

I switched to --storage-driver=overlay and haven't had problems.

I switched to --storage-driver=overlay and haven't had problems.

@joemewes

This comment has been minimized.

Show comment
Hide comment
@joemewes

joemewes Sep 30, 2016

UBUNTU can support AUFS
OSX used AUFS as well (Docker for Mac)

install custom Kernel for Fedora?

Server Version: 1.12.1
Storage Driver: aufs

joemewes commented Sep 30, 2016

UBUNTU can support AUFS
OSX used AUFS as well (Docker for Mac)

install custom Kernel for Fedora?

Server Version: 1.12.1
Storage Driver: aufs

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Sep 30, 2016

Contributor

Overlay is getting better and better.
In 1.12 has overlay2, which requires at least kernel 4.0, but it's faster and uses much fewer inodes than overlay.

note: overlay2 is still the same overlayfs, just a new driver implementation based on new features in overlayfs since the original was released)

Contributor

cpuguy83 commented Sep 30, 2016

Overlay is getting better and better.
In 1.12 has overlay2, which requires at least kernel 4.0, but it's faster and uses much fewer inodes than overlay.

note: overlay2 is still the same overlayfs, just a new driver implementation based on new features in overlayfs since the original was released)

@joemewes

This comment has been minimized.

Show comment
Hide comment
@joemewes

joemewes Sep 30, 2016

@Yajo any reason why you can't switch to OS that natively supports the storage driver you want?

Interesting article about Docker and deviemapper : http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

@Yajo any reason why you can't switch to OS that natively supports the storage driver you want?

Interesting article about Docker and deviemapper : http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/

@virtualfunction

This comment has been minimized.

Show comment
Hide comment
@virtualfunction

virtualfunction Sep 30, 2016

@alercunha FYI AUFS is probably a great way to shoot yourself in the foot in production - you should be aware there are some corner cases where it can screw up royally. (Can't remember what triggers them, but have a feeling it can be when containers share layers)

I've found overlay to perform pretty well and doesn't depend on any FS specific stuff like BtrFS etc. If you are making new containers I would probably go with the overlay2 driver. You ought to be aware Overlay and Overlay2 are not interchangeable in regards to compatibility.

Regardless of driver, I'd advocate where possible using an SSD given container content is liable to be fragmented in more physical locations compared to raw disk access. It'll massively speed up build times too which is important CI-like environments.

@alercunha FYI AUFS is probably a great way to shoot yourself in the foot in production - you should be aware there are some corner cases where it can screw up royally. (Can't remember what triggers them, but have a feeling it can be when containers share layers)

I've found overlay to perform pretty well and doesn't depend on any FS specific stuff like BtrFS etc. If you are making new containers I would probably go with the overlay2 driver. You ought to be aware Overlay and Overlay2 are not interchangeable in regards to compatibility.

Regardless of driver, I'd advocate where possible using an SSD given container content is liable to be fragmented in more physical locations compared to raw disk access. It'll massively speed up build times too which is important CI-like environments.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Sep 30, 2016

Contributor

I'm going to close this as I don't believe there is anything really actionable here.
If someone wants to make a proposal for an improvement to a driver please open a new issue/PR.

Feel free to continue the discussion here.

Contributor

cpuguy83 commented Sep 30, 2016

I'm going to close this as I don't believe there is anything really actionable here.
If someone wants to make a proposal for an improvement to a driver please open a new issue/PR.

Feel free to continue the discussion here.

@cpuguy83 cpuguy83 closed this Sep 30, 2016

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Sep 30, 2016

@virtualfunction thanks for the tip, I've had some bad experience with AUFS in the past but several months ago I decided to give it another shot and I haven't had any issues since then. The storage driver discussion has gone a long way and I haven't yet found a reliable resource pointing to one or the other as the recommended driver regarding Prod environments.

Maybe now there's something new around the block I'm not aware of?

@virtualfunction thanks for the tip, I've had some bad experience with AUFS in the past but several months ago I decided to give it another shot and I haven't had any issues since then. The storage driver discussion has gone a long way and I haven't yet found a reliable resource pointing to one or the other as the recommended driver regarding Prod environments.

Maybe now there's something new around the block I'm not aware of?

@virtualfunction

This comment has been minimized.

Show comment
Hide comment
@virtualfunction

virtualfunction Sep 30, 2016

@alercunha - My general perspective on this (I should disclose I'm heavy docker user and don't come from a Sys/Dev Ops background) is that overlay/overlay2 is generally a good best fit for general purpose stuff if you don't want to actively manage your storage infrastructure (e.g single server / VPS / home server). AUFS is probably best avoided now and was something that promoted in the early days before BtrFS / overlay.

The issue with devicemapper is by default it operates on loopback which will be slow and heavily fragmented on magnetic storage (along with lots of look. Devicemapper (and BtrFS to some degree) really are better in larger scale environments where and needs a bit of planning / active management to operate smoothly.

@alercunha - My general perspective on this (I should disclose I'm heavy docker user and don't come from a Sys/Dev Ops background) is that overlay/overlay2 is generally a good best fit for general purpose stuff if you don't want to actively manage your storage infrastructure (e.g single server / VPS / home server). AUFS is probably best avoided now and was something that promoted in the early days before BtrFS / overlay.

The issue with devicemapper is by default it operates on loopback which will be slow and heavily fragmented on magnetic storage (along with lots of look. Devicemapper (and BtrFS to some degree) really are better in larger scale environments where and needs a bit of planning / active management to operate smoothly.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 30, 2016

Member

Devicemapper (and BtrFS to some degree) really are better in larger scale environments where and needs a bit of planning / active management to operate smoothly.

Overall device mapper is used on RHEL/CentOS based installations, because they don't have support for overlay or aufs. Work is in progress to add official support for overlay(2), and I think it may become the "preferred" option for those platforms as well.

Member

thaJeztah commented Sep 30, 2016

Devicemapper (and BtrFS to some degree) really are better in larger scale environments where and needs a bit of planning / active management to operate smoothly.

Overall device mapper is used on RHEL/CentOS based installations, because they don't have support for overlay or aufs. Work is in progress to add official support for overlay(2), and I think it may become the "preferred" option for those platforms as well.

@alercunha

This comment has been minimized.

Show comment
Hide comment
@alercunha

alercunha Sep 30, 2016

@virtualfunction ok, that's good to know, I will try overlay as you're suggesting and see the results.

It's unfortunate though that this has such a big impact in docker performance and storage issues but there's not a clear/central definition on it. Each system ends up relying on a different driver producing different results.

Of course each installation is different and people must be aware and take care of their own stuff. However it's about time a more professional out-of-the-box solution is provided by Docker so the default installation is one that provide little risk. Also it must be the product responsibility to make people more aware of each driver pitfalls.

@virtualfunction ok, that's good to know, I will try overlay as you're suggesting and see the results.

It's unfortunate though that this has such a big impact in docker performance and storage issues but there's not a clear/central definition on it. Each system ends up relying on a different driver producing different results.

Of course each installation is different and people must be aware and take care of their own stuff. However it's about time a more professional out-of-the-box solution is provided by Docker so the default installation is one that provide little risk. Also it must be the product responsibility to make people more aware of each driver pitfalls.

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah Sep 30, 2016

Member

Also note that overlay2 usually requires kernel 4.x, but support is being back ported to the 3.10 kernel on RHEL/CentOS, in which case you can override the kernel-check to enable it; see #24518, and #24023 for more information. If your kernel supports overlay2, it may be preferable over overlay

Member

thaJeztah commented Sep 30, 2016

Also note that overlay2 usually requires kernel 4.x, but support is being back ported to the 3.10 kernel on RHEL/CentOS, in which case you can override the kernel-check to enable it; see #24518, and #24023 for more information. If your kernel supports overlay2, it may be preferable over overlay

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment