network sandbox join failed: error creating vxlan interface: file exists

[gauravphoenix]

I am running my docker daemon as-

docker -d --kv-store=consul:10.240.222.136:8500 --label=com.docker.network.driver.overlay.bind_interface=eth0 --default-network=overlay:multihost -H tcp://10.240.27.97:2376

and when I run any container, I get following error-

network sandbox join failed: error creating vxlan interface: file exists

I've tried deleting my consul data, and then restarting docker daemon but problem persists.

Seems like the problem occurs when a vxlan interface already exists. How do I go about cleaning up networking related data? I've already tried rm -rf /var/lib/docker/*

Docker info -

root@dockerhost1:/home/gaurav# docker info
Containers: 4
Images: 2
Engine Version: 1.9.0-dev
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 10
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 3.16.0-4-amd64
Operating System: Debian GNU/Linux 8 (jessie)
CPUs: 1
Total Memory: 1.665 GiB
Name: dockerhost1
ID: QHYT:4DUN:6YKL:IJ3Q:CMSU:6F7J:U4SK:2BU4:24WT:3PX2:ZNN2:2G7H
WARNING: No memory limit support
WARNING: No swap limit support
Labels:
 com.docker.network.driver.overlay.bind_interface=eth0
Experimental: true

[mrjana]

@gauravphoenix I am not sure how you ended up in that situation. But to recover from it you can bring down docker and then do tha following:

sudo umount /var/run/docker/netns/*
sudo rm /var/run/docker/netns/*

[gauravphoenix]

@mrjana that worked. Thanks.

[fiunchinho]

I've also came across the same problem. And seems like other people too. Your solution seems to work, but I'm worried something in the documentation leads us to this state.

[melakonda]

In my case it was an issue with docker_gwbridge docker network. This network was missing on the worker node which result following error on that worker node.

Error:
network sandbox join failed: subnet sandbox join failed for "14.2.1.0/24": error creating vxlan interface: file exists

starting container failed: error creating external connectivity network: could not find an available, non-overlapping IPv4 address pool among the defaults to assign to the network

To fix the above errors i have created the docker network manually using below command..
docker network create \ --subnet 172.18.0.0/16 \ --gateway 172.18.0.1 \ -o com.docker.network.bridge.enable_icc=false \ -o com.docker.network.bridge.name=docker_gwbridge \ docker_gwbridge

[yuhao-git-star]

I recover with it.
https://success.docker.com/article/error-network-sandbox-join-failed-during-service-restarts

[hgghyxo]

A server reboot solved for me, don't know, what caused it.

[jamesbdolan]

My swarm service was down due to this issue.

I caused it by doing a yum update of docker while my docker services were still running.

I found a solution here which states that this occurs due to a race condition. dockerd is supposed to delete the vxlan device after a container is stopped.

$ docker service ls
ID             NAME                MODE         REPLICAS   IMAGE                                                                                  PORTS
ze71bkwjaqat   banana_server       replicated   0/1        blah.dkr.ecr.us-east-1.amazonaws.com/banana-api:banana-app-server-ad11cff               *:4030->3030/tcp

$ docker service ps banana_server --no-trunc
ID                          NAME                  IMAGE                             NODE           DESIRED STATE   CURRENT STATE             ERROR                                                                                                                       PORTS
                      
beg13kixclccj0utj59jym24m   banana_server.1       blah.dkr.ecr.us-east-1.amazo...   qa-server-01   Ready           Rejected 3 seconds ago    "network sandbox join failed: subnet sandbox join failed for "10.0.11.0/24": error creating vxlan interface: file exists"
yinps5swi9xf7jmdr27uk5s1u    \_ banana_server.1   blah.dkr.ecr.us-east-1.amazo...   qa-server-01   Shutdown        Rejected 8 seconds ago    "network sandbox join failed: subnet sandbox join failed for "10.0.11.0/24": error creating vxlan interface: file exists"

This was all I needed to do:

• list the vxlan devices that are in a bad state

$ ip -d link show | grep vx
5659: vx-00100b-ripjm: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default
5010: vx-00100c-o0la3: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default

• find which vxlan devices correspond to your docker services (the last 5 characters of the vx-*****-***** ID are a docker network ID)

$ docker network ls | grep -E "ripjm|o0la3"
ripjm8evcjro   banana-net-01                    overlay   swarm
o0la3crbs51r   banana_default            overlay   swarm

• delete the vxlan devices

$ ip link delete vx-00100b-ripjm
$ ip link delete vx-00100c-o0la3

• the docker swarm service should now come up clean

$ docker service ls
ID             NAME                 MODE         REPLICAS   IMAGE                                                                                  PORTS
...
ze71bkwjaqat   banana_server       replicated   1/1        blah.dkr.ecr.us-east-1.amazonaws.com/banana-api:banana-app-server-ad11cff               *:4030->3030/tcp
$

[mr-simonski]

@jamesbdolan Thank you, this perfectly worked for me.

[shencangsheng]

@jamesbdolan Thank you, this perfectly worked for me.

[mario-deriv]

Thanks @jamesbdolan , this worked for us

[MetaSeb]

Hello,

Thanks @jamesbdolan, it worked for us too ! 👍

[psyd01986]

Thanks @jamesbdolan, it worked for me too !