1.13rc1: Cannot create containers with overlay network in docker swarm mode

[gprivitera]

Description
Cannot create a docker container using docker run --net=some-net where some-net is a overlay network created in docker swarm mode.

Steps to reproduce the issue:

1. docker swarm init
2. docker network create --driver=overlay some-net"
3. docker run --net=some-net -d any/image

Describe the results you received:
docker: Error response from daemon: Could not attach to network qiot-dev: rpc error: code = 7 desc = network some-net not manually attachable.

Describe the results you expected:
I did expect the container to start normally and be able to reach it from services and/or other containers attached to the overlay network.

#23901
#25962

Additional information you deem important (e.g. issue happens only occasionally):
Always reproducible.

Output of docker version:

Client:
 Version:      1.13.0-rc1
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   75fd88b
 Built:        Fri Nov 11 22:32:34 2016
 OS/Arch:      darwin/amd64

Server:
 Version:             1.13.0-rc1
 API version:         1.25
 Minimum API version: 1.12
 Go version:          go1.7.3
 Git commit:          75fd88b
 Built:               Fri Nov 11 19:47:07 2016
 OS/Arch:             linux/amd64
 Experimental:        false

Output of docker info:

Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 2
Server Version: 1.13.0-rc1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 16
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: bridge host macvlan null overlay
Swarm: active
 NodeID: a5ccrk6mnsneceaf5rjeaq269
 Is Manager: true
 ClusterID: 7t448hjt5mld2ccb1x7lgsezd
 Managers: 1
 Nodes: 3
 Orchestration:
  Task History Retention Limit: 5
 Raft:
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
 Dispatcher:
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address: 10.0.0.86
 Manager Addresses:
  10.0.0.86:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 8517738ba4b82aff5662c97ca4627e7e4d03b531
runc version: ac031b5bf1cc92239461125f4c1ffb760522bbf2
init version: N/A (expected: v0.13.0)
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.4.0-47-generic
Operating System: Ubuntu 16.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 1
Total Memory: 1.952 GiB
Name: qiot-dev-n0
ID: 2W6D:2RGQ:7JFL:VCWO:EZ7K:AFID:EMO4:ZXCY:MJPW:QLY7:2SRL:HALW
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Username: gprivitera
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Labels:
 provider=amazonec2
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false

Additional environment details (AWS, VirtualBox, physical, etc.):
Using AWS.

[aboch]

When you create the swarm network, you need to specify --attachable if you want to be able to run containers on it.

[gprivitera]

But even if I do that the overlay network is not visible from non-manager nodes, but I suppose this is another issue.

[mrjana]

But even if I do that the overlay network is not visible from non-manager nodes, but I suppose this is another issue.

@gprivitera This is by design. That network will not be visible in worker nodes until you run the first container which attaches to that network.

[gprivitera]

It is not visible even after I attached the first container.

[gprivitera]

$ eval $(docker-machine env instance-n1)
$ docker ps
CONTAINER ID        IMAGE                  COMMAND                  CREATED             STATUS              PORTS                          NAMES
5d61edd12c0f        some/container   "somecommand"   2 minutes ago       Up 2 minutes        80/tcp  container

$ eval $(docker-machine env instance-n2)
$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
a41250fd2dab        bridge              bridge              local
7d692d29b1f4        docker_gwbridge     bridge              local
d9b7e26a39c2        host                host                local
i91dx37b76sz        ingress             overlay             swarm
2672fccd0e09        none                null                local

[mrjana]

Is the container running on that node instance-n2?

[gprivitera]

Nope, it's running on instance-n1.
When I try to create a container on instance-n2 and attach it to the overlay network I get this:

23bc38e0b9521a32637d82938bc7dc5d3318de55642eb4699b9cb3e8b8cd466b
docker: Error response from daemon: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists.

[mrjana]

I am not sure if you were running overlay networks the old way or if you formed a new swarm cluster with new state, there seems to exist a stale vxlan interface with that vxlan id. What kernel version are you on in that node?

[gprivitera]

uname -a
Linux instance-n2 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

I just created both those instances just for this experiment.

[thaJeztah]

@gprivitera you need to create the network with the --attachable option;

docker network create -d overlay --attachable foo

docker run -it --rm --net foo alpine sh

The --attachable option was added to allow docker run to attach to swarm managed overlay networks

[mrjana]

Can you post docker network inspect <some-net> and also sudo ls /var/run/docker/netns output?

[gprivitera]

@mrjana instance-n1 is manager

$ eval $(docker-machine env instance-n1)
$ docker network inspect some-net
[
    {
        "Name": "some-net",
        "Id": "4vuwbddp9vt6mcsgd70vee7ny",
        "Created": "2016-11-15T17:47:55.968491113Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.0.0/24",
                    "Gateway": "10.0.0.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "5d61edd12c0fc64ab78e9df30e433f788ea5eb7f15cf7730891063dfaa55372c": {
                "Name": "some-container",
                "EndpointID": "02ac46e93c8e23cb2356913600c86e7ac42e1933ffdfc94df104e9074836c329",
                "MacAddress": "02:42:0a:00:00:02",
                "IPv4Address": "10.0.0.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": {}
    }
]

$ docker-machine ssh instance-n1
$ sudo ls /var/run/docker/netns
1-4vuwbddp9v  7a7335d23c4a

$ eval $(docker-machine env instance-n2)
$ docker network inspect some-net
[]
Error: No such network: some-net

$ docker-machine ssh instance-n2
$ sudo ls /var/run/docker/netns
1-4vuwbddp9v  1-i91dx37b76  ingress_sbox

[thaJeztah]

@gprivitera have you tried with the --attachable option? From the network inspect output it looks like it's indeed not an attachable network ("Attachable": false,). Creating the network with that option should solve that

[gprivitera]

I did not the first time, then I deleted the network and recreated using the same name with the --attachable option.
I'm recreating the machines from scratch to verify if it solves the problem.

[gprivitera]

Recreated the cluster from scratch..
When I first create the network the attachable option is set to true.
Once I start the first container attached to it it switch back to false, but that's a false positive because I'm able to attach containers to the same network from another node.

$ docker inspect some-net
[
    {
        "Name": "some-net",
        "Id": "l6kzb9rn549ohzddr85y1l3xh",
        "Created": "0001-01-01T00:00:00Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": true,
        "Containers": null,
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4096"
        },
        "Labels": null
    }
]

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
66141572f326        bridge              bridge              local
izfjls5sj73d        some-net                 overlay             swarm
f8a94e3a761f        docker_gwbridge     bridge              local
5a038a52d71f        host                host                local
i2px5flgcuqd        ingress             overlay             swarm
8a5628d0ddde        none                null                local

$ docker network inspect some-net
[
    {
        "Name": "some-net",
        "Id": "izfjls5sj73d0nurwufkimdrq",
        "Created": "2016-11-15T23:44:14.071976075Z",
        "Scope": "swarm",
        "Driver": "overlay",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": [
                {
                    "Subnet": "10.0.1.0/24",
                    "Gateway": "10.0.1.1"
                }
            ]
        },
        "Internal": false,
        "Attachable": false,
        "Containers": {
            "e2abbbe32371622622c3720209591940b0ab4c777425c0252e3ae5561632f613": {
                "Name": "container1",
                "EndpointID": "c1458b003bf3bf6c1670dd30e024541955f4d2e57dc047c8c63b3334b6638a3b",
                "MacAddress": "02:42:0a:00:01:02",
                "IPv4Address": "10.0.1.2/24",
                "IPv6Address": ""
            }
        },
        "Options": {
            "com.docker.network.driver.overlay.vxlanid_list": "4098"
        },
        "Labels": {},
        "Peers": [
            {
                "Name": "instance-n1-b2a02abf42b0",
                "IP": "10.0.0.137"
            }
        ]
    }
]

[thaJeztah]

Good to hear it works!

Once I start the first container attached to it it switch back to false

That looks like a bug @mrjana ^^

[sanimej]

This should be fixed by moby/libnetwork#1545

[gprivitera]

I'm going to close the issue since you guys already identified a bug (and it is not the one the issue started with, which was not a bug).

[ff-foufa]

I have a problem that I can't make connexion between two containers located in the same overlay network "mysql container" and "glpi container"
glpi container needs mysql to be installed. So, where installing (http://localhost : glpi-port) I had access denied to mysql
these are the commands that I used after creating an overlay network colled "afwa":

#docker service create --name my_mysql_server -p 3306:3306 --network afwa --replicas 2 -e MYSQL_ROOT_PASSWORD=root herreraluis/mysql-ubuntu

#docker service create -p 268:80 --name glpi_server --network afwa --replicas 2 -e MYSQL_ROOT_PASSWORD=root herreraluis/docker-glpi

root@ubuntu:/home/afwa# docker network ls
NETWORK ID NAME DRIVER SCOPE
rgds3g2b0qaf afwa overlay swarm
3d72e634e2df bridge bridge local
b74345301711 docker_gwbridge bridge local
927b1820ebfd host host local
iw9cnx6slpyv ingress overlay swarm
a6712b52e00c none null local
root@ubuntu:/home/afwa# docker network inspect rgds3g2b0qaf
[
{
"Name": "afwa",
"Id": "rgds3g2b0qafykx9iviocu794",
"Created": "2017-04-03T08:33:37.481714742-07:00",
"Scope": "swarm",
"Driver": "overlay",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": null,
"Config": [
{
"Subnet": "10.0.0.0/24",
"Gateway": "10.0.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Containers": {
"3070f729e88bf7230ec5642176a8856fb9f4333e4a4024544d1691c2a5e6b47c": {
"Name": "glpi.1.1ziq45js6ewcx4qzwa4z6kk6n",
"EndpointID": "32fdaab3a6668eeef562dee841177a6fb8bca7f39e2ed03cf4ddc98cfc29a6e4",
"MacAddress": "02:42:0a:00:00:06",
"IPv4Address": "10.0.0.6/24",
"IPv6Address": ""
},
"3f6148dc3221e55344b560f5cd17e435fb861ef6b526644238aeed9add85fd1a": {
"Name": "my_mysql_server.2.kakxd9n3w1ude05astlrr6jz4",
"EndpointID": "623d241878624b2390dca39b1a5a7edfc266075d3a89beb407cf91d6b6e9abe9",
"MacAddress": "02:42:0a:00:00:03",
"IPv4Address": "10.0.0.3/24",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.driver.overlay.vxlanid_list": "4097"
},
"Labels": {},
"Peers": [
{
"Name": "ubuntu-d59108f7ed78",
"IP": "192.168.217.154"
},
{
"Name": "ubuntu-9580b0625358",
"IP": "192.168.217.153"
}
]
}
]

[chenjizhao]

I believe most scenarios using docker swarm would like the nodes specified in the same network to work with each other. Therefore I hope the docker development team could set the 'attachable' attribute's default value to true. I and my colleagues really suffered a lot until we find this issue