Ubuntu 16.04 clean install of docker-engine fails to initialize network bridge

[quinnkjones]

On a clean installation of 1.11.2 on Ubuntu 16.04 xenial docker daemon fails to start reporting: "="Error starting daemon: Error initializing network controller: Error creating default "bridge" network: failed to parse pool request for address space "LocalDefault" pool "" subpool "": could not find an available predefined network"

when called via sudo service docker start as listed in the installation page.
When calling via sudo docker daemon at first I get the error Error starting daemon: error initializing graphdriver: "/var/lib/docker" contains several valid graphdrivers: aufs, overlay; Please cleanup or explicitly choose storage driver (-s )

But if I, as suggested by the error message, call sudo docker daemon -s overlay or -s aufs I again receive the error message from above.

I did read through the issues with the same error from earlier versions of docker and attempted the workarounds found there, namely sudo rm -rf /var/lib/docker/network which does not resolve the issue. The other solution my research brought up was to use ifconfig docker0 down, and then restart the bridge, but in this scenario there is no docker0 adapter to disable when I attempt to start the daemon, so I didn't continue down that path.

I considered posting this in #18113, however it is closed due to being reported as fixed in version 1.10.0, so I've created a new one as it would appear to still be an issue, let me know if that was incorrect.

---

BUG REPORT INFORMATION

Docker Version

Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 22:00:43 2016
 OS/Arch:      linux/amd64
Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Output of docker info:

Cannot connect to the Docker daemon. Is the docker daemon running on this host?

Additional environment details (AWS, VirtualBox, physical, etc.):
Physical computer running Ubuntu 16.04
output of uname -a:

Linux quinn-Inspiron-5547 4.4.0-28-generic #47-Ubuntu SMP Fri Jun 24 10:09:13 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Steps to reproduce the issue:

1. Run: sudo service docker start -> receive error
2. Run: systemctl status docker.service

Describe the results you received:
Full log from systemctl status docker.service

● docker.service - Docker Application Container Engine
   Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2016-07-17 15:27:01 EDT; 10min ago
     Docs: https://docs.docker.com
  Process: 30547 ExecStart=/usr/bin/docker daemon -s overlay -H fd:// (code=exited, status=1/FAILURE)
 Main PID: 30547 (code=exited, status=1/FAILURE)

Jul 17 15:27:00 quinn-Inspiron-5547 systemd[1]: Starting Docker Application Container Engine...
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.876634901-04:00" level=info msg="previous instance of containerd still alive (14264)"
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.940600196-04:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.946582731-04:00" level=info msg="Firewalld running: false"
Jul 17 15:27:01 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:01.024951094-04:00" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: failed to parse pool request for address space \"LocalDefault\" pool \"\" subpool \"\": could not find an available predefined network"
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: Failed to start Docker Application Container Engine.
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Unit entered failed state.
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Failed with result 'exit-code'.

relevant output of journalctl -xe:

Jul 17 15:27:00 quinn-Inspiron-5547 sudo[30445]:    quinn : TTY=pts/5 ; PWD=/home/quinn ; USER=root ; COMMAND=/usr/sbin/service docker start
Jul 17 15:27:00 quinn-Inspiron-5547 sudo[30445]: pam_unix(sudo:session): session opened for user root by quinn(uid=0)
Jul 17 15:27:00 quinn-Inspiron-5547 systemd[1]: Listening on Docker Socket for the API.
-- Subject: Unit docker.socket has finished start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.socket has finished starting up.
-- 
-- The start-up result is done.
Jul 17 15:27:00 quinn-Inspiron-5547 systemd[1]: Starting Docker Application Container Engine...
-- Subject: Unit docker.service has begun start-up
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has begun starting up.
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.876634901-04:00" level=info msg="previous instance of containerd still alive (14264)"
Jul 17 15:27:00 quinn-Inspiron-5547 audit[30557]: AVC apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=30557 comm="apparmor_parser"
Jul 17 15:27:00 quinn-Inspiron-5547 kernel: audit: type=1400 audit(1468783620.901:31): apparmor="STATUS" operation="profile_replace" profile="unconfined" name="docker-default" pid=30557 comm="apparmor_parser"
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.940600196-04:00" level=info msg="Graph migration to content-addressability took 0.00 seconds"
Jul 17 15:27:00 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:00.946582731-04:00" level=info msg="Firewalld running: false"
Jul 17 15:27:01 quinn-Inspiron-5547 docker[30547]: time="2016-07-17T15:27:01.024951094-04:00" level=fatal msg="Error starting daemon: Error initializing network controller: Error creating default \"bridge\" network: failed to parse pool request for address space \"LocalDefault\" pool \"\" subpool \"\": could not find an available predefined network"
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Main process exited, code=exited, status=1/FAILURE
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: Failed to start Docker Application Container Engine.
-- Subject: Unit docker.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
-- 
-- Unit docker.service has failed.
-- 
-- The result is failed.
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Unit entered failed state.
Jul 17 15:27:01 quinn-Inspiron-5547 systemd[1]: docker.service: Failed with result 'exit-code'.

Describe the results you expected:

No error code and a running docker daemon preferably.

Additional information you deem important (e.g. issue happens only occasionally):
Occurs consistently despite multiple reboots, attempts etc.

[aboch]

@quinnkjones
Please check moby/libnetwork#779
Check if there is any route in your sytem which would mask all the predefined address pools from which docker picks a subnet for the default bridge network (docker0). Usually vpn routes would do so.

[thaJeztah]

@quinnkjones you mention that you got the "Please cleanup or explicitly choose storage driver" message;

graphdriver: "/var/lib/docker" contains several valid graphdrivers: aufs, overlay; Please cleanup or explicitly choose storage driver (-s )
But if I, as suggested by the error message, call sudo docker daemon -s overlay or -s aufs I again receive the error message from above.

Did you only start the daemon manually with that option, or also update the daemon configuration (either by using a systemd drop-in file, or by using a daemon.json configuration file?)

sudo docker daemon -s overlay will only start a daemon manually, but will not update the configuration for the "service". If you tried to run the daemon manually while the service was still running, that may also result in two daemons running at the same time.

Other than that, please check the link that @aboch provided

[quinnkjones]

Thanks to you both for replying, I've only just now gotten off of work, so it's time to reply!
@aboch
I didn't have anything active at the time or now so I don't think this was it.
@thaJeztah
I've also modified the service config to run with -s overlay option.

Better yet! it appears the problem has solved itself in classic linux style by turning it off and on again, as suggested in the installation instructions I used the sudo reboot command to reboot between installing the headers and installing the docker-engine; and in addition I rebooted in this fashion multiple time when it wasn't working. But now after a hard boot, no problems; I can only assume there must be some functional difference at least for my setup that causes the former not to work.

[thaJeztah]

Thanks! Good to hear it's resolved. I'll go ahead and close this, but comment if you encounter this again 👍

[quinnkjones]

So apparently this happens intermittently as it is now happening again!
Here is the output of route:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.4.128.1      0.0.0.0         UG    600    0        0 wlp2s0
10.0.0.0        10.4.128.1      255.0.0.0       UG    600    0        0 wlp2s0
10.4.128.0      *               255.255.240.0   U     600    0        0 wlp2s0
157.182.0.0     10.4.128.1      255.255.0.0     UG    600    0        0 wlp2s0
157.182.94.70   10.4.128.1      255.255.255.255 UGH   600    0        0 wlp2s0
172.16.0.0      10.4.128.1      255.240.0.0     UG    600    0        0 wlp2s0
192.168.0.0     10.4.128.1      255.255.0.0     UG    600    0        0 wlp2s0

My guess now is that the main difference between the set of times when I could run docker and the set of times I couldn't is that I can run docker in my apartment (or more accurately when I'm on my apartments very simple network), but when I'm at work (rather on a university buildings network) the routes are different enough to cause the error. Is the problem the A class netmask on 10.0.0.0? or the ambitious default gateway?

[quinnkjones]

Even more support for my theory of the network people being responsible I had a meeting in another building, and I tried again to start the daemon service and it worked, despite being on the same enterprise network, so it must be something with the gateway setup in the building in which I work.
The output of route here is:

Kernel IP routing table
Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
default         10.255.64.1     0.0.0.0         UG    600    0        0 wlp2s0
1.1.1.100       10.255.64.1     255.255.255.255 UGH   600    0        0 wlp2s0
10.255.64.0     *               255.255.192.0   U     600    0        0 wlp2s0
link-local      *               255.255.0.0     U     1000   0        0 wlp2s0
172.17.0.0      *               255.255.0.0     U     0      0        0 docker0

Though interestingly now that I've got that to occur when I attempt to run a "hello world" container:

 docker run ubuntu /bin/echo 'Hello world'

I get the error:

 docker: Cannot connect to the Docker daemon. Is the docker daemon running on this host?.
See 'docker run --help'.

Which is odd, as I haven't changed the host setting in the systemd service command. It is still set as -H fd://. I've attempted to call docker run with -H fd:// but no luck.

[mario-areias]

@quinnkjones

I just faced the same issue. In my case, it was my VPN who was causing trouble. As soon as I disconnected, the installation worked. Then I connected the VPN again and run the docker hello-world command and it works. Not sure if that helps you, but it is worth try it.

[marcellmars]

same here with bitmask (vpn) running..

Docker version 1.12.3, build 6b644ec
Debian unstable, kernel 4.7.8-1 (2016-10-19) x86_64 GNU/Linux

turning off bitmask helped docker systemd service starting...

[slayer]

moby/libnetwork#779

I locked this issue because it become a bucket where to drop "I have this issue too" comments, while the root cause of the issue has been explained long ago and workarounds have been provided.

Ha ha. Thread locking is a great solution of problem. No complains - no problem.

Yes, I have same problem

[mavenugo]

@slayer did you try the workaround suggested in that issue : moby/libnetwork#779 (comment) ?
and do you have any feedback on that ?

[slayer]

@mavenugo sure. And it works for me. Thanks.
But. I manage many servers with openvpn (and my laptop too).
And to fix the problem, I need to do some not obvious actions on each system, it is not convenient.

IMHO that is not a solution. Its a dirty workaround.

[thaJeztah]

@slayer you can set the option through the --bip daemon flag (which can be set, e.g. in a sytemd drop-in file), or through the daemon.json configuration file, for example {"bip": "192.168.1.5/24"}. Docker tries to do as much configuration automatically (to get things going), but to run docker in production, configuration will be needed in most cases.

[slayer]

@thaJeztah Thanks. I've already added bip config to /etc/default/docker.

What you think about ignoring /1 (and maybe /2 too) masks in network selection decision ?

[aboch]

@slayer I think your suggestion is a good compromise. Feel free to push a PR in libnetwork.
I guess you would modify the loop at https://github.com/docker/libnetwork/blob/master/netutils/utils_linux.go#L32 to continue if the ones in network.Dst.Mask.Size() is less or equal to 2.
Thanks.

[sanimej]

@aboch I think a better fix would be to check only the interface subnets on the host for the conflict. wdyt ?