Docker Daemon Aufs Mount Panic #21078

Closed
dadgar opened this Issue Mar 9, 2016 · 5 comments

Comments

Projects
None yet
7 participants
@dadgar

dadgar commented Mar 9, 2016

Output of docker version:

Client:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:37:01 2016
 OS/Arch:      linux/amd64
Server:
 Version:      1.10.2
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   c3959b1
 Built:        Mon Feb 22 21:37:01 2016
 OS/Arch:      linux/amd64

Output of docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 3
Server Version: 1.10.2
Storage Driver: aufs
 Root Dir: /opt/docker/data/aufs
 Backing Filesystem: extfs
 Dirs: 3
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.19.0-43-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 29.45 GiB
Name: c1m-nomad-client-us-central1-b-n1-standard-8-1-1mu9
ID: QK7Z:NE6P:BKB6:L6EJ:CYC4:EULW:46AA:FMGN:35V7:GYJ3:E2VV:V2CV
WARNING: No swap limit support

Provide additional environment details (AWS, VirtualBox, physical, etc.):
Environment: GCE
Machine Type: n1-standard-8

List the steps to reproduce the issue:

  1. Running hundreds of containers on the same docker daemon in a very short time frame

This is fairly reproducible, on a 500 node cluster, 8 nodes had this panic.

Describe the results you received:

Docker daemon panics. Logs attached.

docker_2.txt
docker_3.txt
docker.txt

@HackToday

This comment has been minimized.

Show comment
Hide comment
@HackToday

HackToday Mar 10, 2016

Contributor

From above comments, the case is create and delete containers repeatedly right ? Parallel test?

Contributor

HackToday commented Mar 10, 2016

From above comments, the case is create and delete containers repeatedly right ? Parallel test?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Mar 10, 2016

Contributor

This is probably because of https://github.com/docker/docker/blob/v1.10.2/daemon/graphdriver/aufs/aufs.go#L230
Modification is happening while not protected by a mutex.
This is most likely fixed in master since it is now protected: https://github.com/docker/docker/blob/master/daemon/graphdriver/aufs/aufs.go#L232

Contributor

cpuguy83 commented Mar 10, 2016

This is probably because of https://github.com/docker/docker/blob/v1.10.2/daemon/graphdriver/aufs/aufs.go#L230
Modification is happening while not protected by a mutex.
This is most likely fixed in master since it is now protected: https://github.com/docker/docker/blob/master/daemon/graphdriver/aufs/aufs.go#L232

@dadgar

This comment has been minimized.

Show comment
Hide comment
@dadgar

dadgar Mar 10, 2016

@HackToday it is mainly a batch of creates and then minutes later a batch of deletes

dadgar commented Mar 10, 2016

@HackToday it is mainly a batch of creates and then minutes later a batch of deletes

@kenorb

This comment has been minimized.

Show comment
Hide comment

kenorb commented May 15, 2017

@thaJeztah

This comment has been minimized.

Show comment
Hide comment
@thaJeztah

thaJeztah May 15, 2017

Member

Let me close this issue, because this was most likely fixed (see #21078 (comment)), and we received no other reports of this since

Member

thaJeztah commented May 15, 2017

Let me close this issue, because this was most likely fixed (see #21078 (comment)), and we received no other reports of this since

@thaJeztah thaJeztah closed this May 15, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment