Containers are not started in parallel

[sakserv]

Description

Containers are not started in parallel when issuing multiple concurrent "docker run" commands for the same image.

I expect this has to have been discussed in the past, but my searches have turned up empty. Spent the last two days on IRC attempting to get more details to no avail.

Steps to reproduce the issue:

1. Issue "docker run" for 10 containers in the background, printing the start time.
2. Wait for "docker run" to exit/return.
3. Print the timestamp when "docker run" completes.

Describe the results you received:
The first docker run completes in ~2 seconds, whereas the tenth docker run completes in ~25 seconds.

Describe the results you expected:
Given enough resources on the host, all of these should start in parallel. All containers should start within ~2 seconds.

Additional information you deem important (e.g. issue happens only occasionally):

Gist with the timings:
https://gist.github.com/sakserv/bc55f10fd35801bb374e8b8401bf07bb

Output of docker version:

Client:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.12.5
 API version:  1.24
 Go version:   go1.6.4
 Git commit:   7392c3b
 Built:        Fri Dec 16 02:23:59 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 9
 Running: 5
 Paused: 0
 Stopped: 4
Images: 48
Server Version: 1.12.5
Storage Driver: devicemapper
 Pool Name: vg01-docker--pool
 Pool Blocksize: 524.3 kB
 Base Device Size: 268.4 GB
 Backing Filesystem: xfs
 Data file:
 Metadata file:
 Data Space Used: 41.2 GB
 Data Space Total: 5.63 TB
 Data Space Available: 5.589 TB
 Metadata Space Used: 16.17 MB
 Metadata Space Total: 16.98 GB
 Metadata Space Available: 16.96 GB
 Thin Pool Minimum Free Space: 563 GB
 Udev Sync Supported: true
 Deferred Removal Enabled: false
 Deferred Deletion Enabled: false
 Deferred Deleted Device Count: 0
 Library Version: 1.02.107-RHEL7 (2015-12-01)
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: null host bridge overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Security Options: seccomp
Kernel Version: 3.10.0-327.13.1.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
CPUs: 32
Total Memory: 251.6 GiB
Name: foo.example.com
ID: 7DE4:CNIR:C3A5:HYGP:NUJN:DUJ2:3LYC:5TBZ:37EA:WWTT:UZPT:XLV3
Docker Root Dir: /grid/0/docker
Debug Mode (client): false
Debug Mode (server): true
 File Descriptors: 38
 Goroutines: 40
 System Time: 2016-12-29T15:50:33.050576556Z
 EventsListeners: 0
Registry: https://index.docker.io/v1/
WARNING: bridge-nf-call-iptables is disabled
WARNING: bridge-nf-call-ip6tables is disabled
Insecure Registries:
 127.0.0.0/8

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical node - centos 7.2

[sakserv]

@thaJeztah - Is this limitation already known? I've been spending the morning trying to narrow down the bottleneck after it was suggested on IRC that it could be containerd related. If this issue is already known, I'll save myself the effort of gathering additional debug details. :) Thanks.

Regarding the containerd comment above, fwiw, I tried downgrading to docker 1.10.3, pre-containerd, and I'm still seeing the same behavior.

[cpuguy83]

@sakeven Try a different storage driver. Not only is devicemapper the slowest, and setup of the storage driver is (probably) the slowest step in the process, this part also has to by synchronized to avoid data races.

[sakeven]

@cpuguy83 mentioned wrong person.

@sakserv