Failed to create the network with overlay driver type

[wenchma]

Description of problem:

I follow the network doc: https://github.com/docker/docker/blob/master/docs/userguide/networking/work-with-networks.md , but I cannot create overlay network

ubuntu@docker:$ docker network create -d overlay --subnet=192.168.0.0/16 --subnet=192.170.0.0/16 --gateway=192.168.0.100 --gateway=192.170.0.100 --ip-range=192.168.1.0/24 --aux-address a=192.168.1.5 --aux-address b=192.168.1.6 --aux-address a=192.170.1.5 --aux-address b=192.170.1.6 my-multihost-network
Error response from daemon: error getting pools config from store during init: store for address space GlobalDefault not found
ubuntu@docker:$ docker network ls
NETWORK ID NAME DRIVER
9c6be6eb17fc bridge bridge
6be768fdb1d7 none null
e50bf9a4e655 host host
dcec95685b86 simple-network bridge

docker version :

Client:
 Version:      1.9.0-rc5
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9318004
 Built:        Tue Nov  3 05:59:27 UTC 2015
 OS/Arch:      linux/amd64

Server:
 Version:      1.9.0-rc5
 API version:  1.21
 Go version:   go1.4.2
 Git commit:   9318004
 Built:        Tue Nov  3 05:59:27 UTC 2015
 OS/Arch:      linux/amd64 

docker info :

Containers: 0
Images: 0
Server Version: 1.9.0-rc5
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 0
 Dirperm1 Supported: false
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.13.0-43-generic
Operating System: Ubuntu 14.04.1 LTS
CPUs: 4
Total Memory: 15.67 GiB
Name: docker
ID: UT5D:M5ZY:IEXQ:6LCU:YX6J:ZMWM:BLNJ:MTXH:PRHO:ETVF:3FHH:6LC4
WARNING: No swap limit support

uname -a :

Linux docker 3.13.0-43-generic #72-Ubuntu SMP Mon Dec 8 19:35:06 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):OpenStack Cloud instance

[j4fm]

Same error here with Docker 1.9 rc3 & rc4 and Azure when using Ubuntu Linux Kernel versions higher than 3.13. docker-machine create against azure also times out after 60 retries even though the Azure VM is created okay. Same error after that with "Error response from daemon: error getting pools config from store during init: store for address space GlobalDefault not found" from "docker network create -d overlay networkname"

[cpuguy83]

You need at least kernel 3.16, and you need to specify --cluster-advertise and --cluster-store on the daemon opts.

[j4fm]

All three of those requirements have been met when I get the problem.

[mavenugo]

@jamesmeen if you have all the configurations done correctly, then docker info output such include the cluster store and advertise output.

Example :

Cluster store: etcd://192.168.33.10:42000/newroot
Cluster advertise: 192.168.33.12:0

Can you please double check your configurations ?

[j4fm]

Those do not appear in docker info. I'm just using consul as per the document.

Oddly, enough the exact same docker-machine create command succeeds in creating an Azure VM when the kernel is 3.13. Also, with kernel 3.13, the "docker networking create" command works too and I can see the overlay networks created and discovered across hosts. It fails to start a container against the network though.

With 3.16 onwards, the same commands work less than with 3.13 kernel.

[j4fm]

Actually, I have found my specific problem actually relates to docker-machine not supported Ubuntu 15.04 or newer which has the newer kernels included. Ubuntu 15.04 switched to systemd so docker starts with no configuration, regardless of parameters passed into docker-machine.

Docker Machine issue #1891

Thank you

[wenchma]

did nothing when creating machine, why to report resources were removed ?

ubuntu@ubuntu14:~$ docker-machine create -d virtualbox mh-keystore
Creating VirtualBox VM...
Creating SSH key...
Starting VirtualBox VM...
Starting VM...
Error creating machine: Too many retries.  Last error: Maximum number of retries (60) exceeded
You will want to check the provider to make sure the machine and associated resources were properly     removed.
ubuntu@ubuntu14:~$ docker-machine ls
error getting URL for host mh-keystore: exit status 255
error getting URL for host mh-keystore: exit status 255
error determining if host is active for host mh-keystore: exit status 255
NAME          ACTIVE   DRIVER       STATE     URL   SWARM
mh-keystore            virtualbox   Running

[wenchma]

/user/local/bin/docker daemon -D -g /var/lib/docker -H unix:// -H tcp:0.0.0.0:2376 --label provider=virtualbox --cluster-store=consul://192.168.99.100:8500 --cluster-advertise=eth1:2376 --tlsverify --tlskey=/var/lib/boot2docker/server-key.pem --tlscert=/var/lib/boot2docker/server.pem --tlscacert=/var/lib/boot2docker/ca.pem -s aufs

$ docker network create -d overlay --subnet=192.168.0.0/16 --subnet=192.170.0.0/16 -- gateway=192.168.0.100 --gateway=192.170.0.100 --ip-range=192.168.1.0/24 --aux-address a=192.168.1.5 --aux-address b=192.168.1.6 --aux-address a=192.170.1.5 --aux-address b=192.170.1.6 my-multihost-network
376f9b5a2a35a530a01cbe411fa32f149ff39d94ec10a16f7fd389e5954cdf35
$ docker network ls
NETWORK ID          NAME                        DRIVER
d3477b5cdac2        mhs-demo0/host              host                
13d65aaa86c0        mhs-demo0/docker_gwbridge   bridge              
25e6a3f7981a        my-net                      overlay             
32806487946a        mhs-demo0/none              null                
cf643507b9c1        mhs-demo1/host              host                
376f9b5a2a35        my-multihost-network        overlay             
fcddfdf864bd        mhs-demo0/bridge            bridge              
d6413651b16a        mhs-demo1/none              null                
f5ff60ef51f4        mhs-demo1/docker_gwbridge   bridge              
b3b6546cb1e9        mhs-demo1/bridge            bridge              
$ docker network inspect my-multihost-network
[
    {
        "Name": "my-multihost-network",
        "Id": "376f9b5a2a35a530a01cbe411fa32f149ff39d94ec10a16f7fd389e5954cdf35",
        "Scope": "global",
        "Driver": "overlay",
        "IPAM": {
            "Driver": "default",
            "Config": [
                {
                    "Subnet": "192.170.0.0/16",
                   "Gateway": "192.170.0.100",
                   "AuxiliaryAddresses": {
                        "a": "192.170.1.5",
                        "b": "192.170.1.6"
                   }
                },
                {
                    "Subnet": "192.168.0.0/16",
                    "IPRange": "192.168.1.0/24",
                    "Gateway": "192.168.0.100"
                }
            ]
        },
        "Containers": {},
        "Options": {}
    }
]
$ docker run -itd --name=container2 --net=my-multihost-network --env="constraint:node==mhs-demo1" busybox
$ docker inspect --format '{{json .NetworkSettings.Networks}}' mhs-demo1/container2
{"my-multihost-network": {"EndpointID":"55c356136901dd3168b16512a0ec8b546af75d407bdd8b7668290e06d7f91570","Gateway":"","IPAddress":"192.170.0.1","IPPrefixLen":16,"IPv6Gateway":"","GlobalIPv6Address":"","GlobalIPv6PrefixLen":0,"MacAddress":"02:42:c0:aa:00:01"}}

[wenchma]

@mavenugo for my above comment, I create an overlay network with multi subnets, then create a container with specifying the overlay network, the container only obtain the first subnet, why did it get one subnet ? not two ? ❓

[wenchma]

resolved and close it, thanks!

[brikeyes]

I am not sure what the resolution is here ,? would docker 1.11 multi host networking not be supported Ubuntu 15.10

[brikeyes]

I restarted the host , removed the consul container , rebuilt the container with
"docker run -d -p 8500:8500 -h consul --name consul progrium/consul -server –bootstrap"
and it works now