[1.10] Removing a network during shutdown leaves Docker inoperable

[bboreham]

Description of problem:

If you have a Docker Network plugin that tries to remove its network on shutdown, and you restart the Docker daemon, it then restarts continually.

docker version:

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:36:33 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:36:33 2016
 OS/Arch:      linux/amd64

docker info:

Containers: 0
 Running: 0
 Paused: 0
 Stopped: 0
Images: 631
Server Version: 1.10.0
Storage Driver: aufs
 Root Dir: /home/docker/aufs
 Backing Filesystem: extfs
 Dirs: 564
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: null host bridge
Kernel Version: 3.13.0-71-generic
Operating System: Ubuntu 14.04.3 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 5.829 GiB
Name: bbvlx1
ID: IVMU:DIYM:GNIF:DNSW:GCHY:42Z5:R573:MDYB:YWOK:W57O:P4WY:2LQS
Username: bboreham
Registry: https://index.docker.io/v1/
WARNING: No swap limit support

uname -a:

Linux bbvlx1 3.13.0-71-generic #114-Ubuntu SMP Tue Dec 1 02:34:22 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

VirtualBox

How reproducible:

100%

Steps to Reproduce:

Using Weave Net 1.4.3:

$ weave launch-router && weave launch-plugin
3c19d3e02d07e13967690044b27e6029bd3a6d0f2623984200a0e4ff8f3a355d
e183a8e9973cc1331627386ad34158602cb13a3f1bf27b28a0793a8a47871b03
$ sudo service docker restart

Actual Results:

Docker log from the restart onwards:

INFO[0017] Processing signal 'terminated'               
ERRO[0017] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0017] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
INFO[0027] Container 975bf48d15c0d8128e47bf27d1ed9615226f3012de4fea7233e694629b844a71 failed to exit within 10 seconds of SIGTERM - using the force 
WARN[0027] Unable to connect to plugin: /run/docker/plugins/weavemesh.sock, retrying in 1s 
ERRO[0027] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0027] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0027] Handler for GET /v1.21/containers/weave/json returned error: write unix /var/run/docker.sock->@: write: broken pipe 
2016/02/09 10:23:21 http: multiple response.WriteHeader calls
WARN[0028] Unable to connect to plugin: /run/docker/plugins/weavemesh.sock, retrying in 2s 
ERRO[0028] Error during layer Store.Cleanup(): device or resource busy 
Waiting for /var/run/docker.sock
INFO[0000] [graphdriver] using prior storage driver "aufs" 
/var/run/docker.sock is up
INFO[0000] Graph migration to content-addressability took 0.00 seconds 
INFO[0000] Firewalld running: false                     
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
ERRO[0001] Error during layer Store.Cleanup(): device or resource busy 
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use 
/var/run/docker.sock is up
INFO[0000] [graphdriver] using prior storage driver "aufs" 
INFO[0000] Graph migration to content-addressability took 0.00 seconds 
INFO[0000] Firewalld running: false                     
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
ERRO[0001] Error during layer Store.Cleanup(): device or resource busy 
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use 
/var/run/docker.sock is up
INFO[0000] [graphdriver] using prior storage driver "aufs" 
INFO[0000] Graph migration to content-addressability took 0.00 seconds 
INFO[0000] Firewalld running: false                     
WARN[0000] Could not get list of networks during endpoint cleanup: could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
ERRO[0000] could not find endpoint count key docker/network/v1.0/endpoint_count/285a26bf80dc437fe4c12a9ba7890990d466e26da81d1e13216cea35e1cd93cb/ for network weave while listing: Key not found in store 
INFO[0000] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 

ad infinitum

Expected Results:

Calm restart of Docker process; perhaps one or two error messages noting that the plugin did something inadvisable.

Additional info:

The crux of the problem seems to be that the Weave Net plugin attempts a RemoveNetwork via the API in the middle of the shutdown sequence. Removing this call avoids the issue.

The same plugin code didn't break Docker 1.9.

[cpuguy83]

ping @mavenugo

[mavenugo]

@bboreham restarting continuously is due to the systemd configuration on failed start. So the question here is the reason for the daemon failure during the start.

ERRO[0001] Error during layer Store.Cleanup(): device or resource busy 
FATA[0001] Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to allocate gateway (172.17.0.1): Address already in use 

I think I have an idea.... We will push a fix soon.

[mavenugo]

This issue is not full resolved. reopening.