Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945

brettdh · 2016-02-11T15:15:08Z

(Similar, but closed, issues: #562, #751)

Very occasionally, I see this error message when starting a container in my swarm:

Error response from daemon: Cannot start container <container hash>: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists

This error persists until I reboot the docker hosts. A comment on #751 suggested that restarting iptables would suffice; I have not tried this yet. I also have tried the solution mentioned in #562 previously, and I believe that worked as well, but I cannot remember for sure.

docker version:

Client:
 Version:      1.10.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   590d5108
 Built:        Thu Feb  4 18:18:11 2016
 OS/Arch:      darwin/amd64

Server:
 Version:      swarm/1.1.0
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   a0fd82b
 Built:        Thu Feb  4 08:55:18 UTC 2016
 OS/Arch:      linux/amd64

docker info:

Note: this was not captured when the error was occurring. If it happens again, I will comment with the info.

Containers: 32
 Running: 29
 Paused: 0
 Stopped: 3
Images: 222
Role: primary
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 4
 druid-vms.historical1: <host3>:2376
  └ Status: Healthy
  └ Containers: 4
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 8.188 GiB
  └ Labels: executiondriver=native-0.2, historical=yes, kernelversion=3.16.0-30-generic, operatingsystem=Ubuntu 14.04.2 LTS, provider=generic, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-02-11T15:00:03Z
 druid-vms.historical2: <host4>:2376
  └ Status: Healthy
  └ Containers: 4
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 8.188 GiB
  └ Labels: executiondriver=native-0.2, historical=yes, kernelversion=3.16.0-30-generic, operatingsystem=Ubuntu 14.04.2 LTS, provider=generic, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-02-11T15:00:24Z
 druid-vms.ingestion1: <host2>:2376
  └ Status: Healthy
  └ Containers: 5
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 8.188 GiB
  └ Labels: executiondriver=native-0.2, ingestion=yes, kernelversion=3.16.0-30-generic, operatingsystem=Ubuntu 14.04.2 LTS, provider=generic, storagedriver=aufs
  └ Error: (none)
  └ UpdatedAt: 2016-02-11T14:59:45Z
 druid-vms.queen: <host1>:2376
  └ Status: Healthy
  └ Containers: 19
  └ Reserved CPUs: 0 / 2
  └ Reserved Memory: 0 B / 8.188 GiB
  └ Labels: broker=yes, consul=yes, deployment-master=yes, executiondriver=native-0.2, kernelversion=3.16.0-30-generic, metadata=yes, operatingsystem=Ubuntu 14.04.2 LTS, provider=generic, storagedriver=aufs, swarm-master=yes
  └ Error: (none)
  └ UpdatedAt: 2016-02-11T15:00:22Z
Plugins: 
 Volume: 
 Network: 
Kernel Version: 3.16.0-30-generic
Operating System: linux
Architecture: amd64
CPUs: 8
Total Memory: 32.75 GiB
Name: druid-vms.queen
Http Proxy: <snip>
Https Proxy: <snip>
No Proxy: <snip>

uname -a:

Local: Darwin <hostname> 15.3.0 Darwin Kernel Version 15.3.0: Thu Dec 10 18:40:58 PST 2015; root:xnu-3248.30.4~1/RELEASE_X86_64 x86_64
Docker hosts: Linux <hostname> 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:43:14 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

Docker Swarm, 4 pre-existing VMs, generic driver

How reproducible:

Intermittent; not reliably reproducible (and I wouldn't begin to know how to try). Seems to occur sometime after destroying and re-creating the swarm using docker-machine.

The text was updated successfully, but these errors were encountered:

tomashejatko · 2016-02-25T14:35:34Z

Happens to us pretty random (or not in known/reproducible way) also.
docker version:

Client:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.1
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   9e83765
 Built:        Thu Feb 11 19:27:08 2016
 OS/Arch:      linux/amd64

Docker info from third node of our swarm cluster, only one that have problem atm.

Containers: 12
 Running: 2
 Paused: 0
 Stopped: 10
Images: 15
Server Version: 1.10.1
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 297
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: overlay host bridge null
Kernel Version: 4.2.0-27-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 15.67 GiB
Name: host3
ID: V3UE:GFTF:TZ2B:2IOU:Q7YW:FD6A:PXOK:NHUT:V6DT:ZT6T:5H4W:22D2
WARNING: No swap limit support
Cluster store: etcd://<host1>:2379,<host2>:2379,<host3>:2379/_pa

Cluster advertise: :4243

EDIT :
I did just copypaste from #751, but Ubuntu does not have Sys V init script for iptables, so I just restarted docker and problem is gone now :

 [root@<host3> ~]# service docker stop; service iptables restart; service docker start
docker stop/waiting
iptables: unrecognized service
docker start/running, process 13824

brettdh · 2016-03-02T18:21:24Z

Saw this again today, after destroying and re-creating my --driver generic docker machines. No meaningful changes in the docker info output; just UpdatedAt and Containers counts.

Oh, it now says Server Version: swarm/1.1.2 (Client and server docker version still 1.10.0).

Restarting docker didn't work for me like it did for @arteal, so I ended up rebooting all the hosts again.

tomashejatko · 2016-03-02T21:56:34Z

Happened to us again, restarting docker didn't help, stopping docker, flushing iptables and starting docker did not help. Had to reboot that machine

frntn · 2016-03-16T14:29:25Z

Same as @arteal for me
Unable to identify the root cause so unable to reproduce on purpose.
Seems to happen randomly.

I'll try to get more info next time...

docker version:

Client:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.10.3
 API version:  1.22
 Go version:   go1.5.3
 Git commit:   20f81dd
 Built:        Thu Mar 10 15:54:52 2016
 OS/Arch:      linux/amd64

docker info:

Containers: 1
 Running: 1
 Paused: 0
 Stopped: 0
Images: 9
Server Version: 1.10.3
Storage Driver: aufs
 Root Dir: /data/.graph/var/lib/releases/20160316_1458121754/aufs
 Backing Filesystem: extfs
 Dirs: 210
 Dirperm1 Supported: true
Execution Driver: native-0.2
Logging Driver: json-file
Plugins: 
 Volume: local
 Network: overlay null host bridge
Kernel Version: 3.16.0-67-generic
Operating System: Ubuntu 14.04.1 LTS
OSType: linux
Architecture: x86_64
CPUs: 2
Total Memory: 1.421 GiB
Name: xxxxxxx
ID: YPJJ:UE7N:FYIZ:AOXA:EXVL:7B7I:67U7:YUPK:NOZB:WMLJ:J4YH:GWOV
WARNING: No swap limit support
Labels:
 project=xxxxxx
 env=test
 type=bastion
Cluster store: consul://x.x.x.x:8502/network
Cluster advertise: z.z.z.z:2375

uname -a:

Linux xxxxxxxx 3.16.0-67-generic #87~14.04.1-Ubuntu SMP Fri Mar 11 00:26:02 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux

Environment details (AWS, VirtualBox, physical, etc.):

Physical

larsla · 2016-03-20T16:41:57Z

I just experienced this issue.

First i got:
500 Internal Server Error: subnet sandbox join failed for "10.0.0.0/24": vxlan interface creation failed for subnet "10.0.0.0/24": failed in prefunc: failed to set namespace on link "vx-000104-52662": invalid argument

Then after a few tries it switched to saying:
500 Internal Server Error: subnet sandbox join failed for "10.0.4.0/24": error creating vxlan interface: file exists

Doing a "service docker stop; service docker start" helped.

docker info

Containers: 34
Running: 27
Paused: 0
Stopped: 7
Images: 35
Server Version: swarm/1.1.3
Role: replica
Primary: 10.42.0.232:4000
Strategy: spread
Filters: health, port, dependency, affinity, constraint
Nodes: 3
swarm04: 10.42.0.250:2375
└ Status: Healthy
└ Containers: 12
└ Reserved CPUs: 0 / 8
└ Reserved Memory: 4 GiB / 32.91 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.4.4-std-3, operatingsystem=Ubuntu 14.04.4 LTS, storagedriver=btrfs
└ Error: (none)
└ UpdatedAt: 2016-03-20T19:04:44Z
swarm05: 10.42.0.232:2375
└ Status: Healthy
└ Containers: 11
└ Reserved CPUs: 0 / 8
└ Reserved Memory: 4 GiB / 32.91 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.4.4-std-3, operatingsystem=Ubuntu 14.04.4 LTS, storagedriver=btrfs
└ Error: (none)
└ UpdatedAt: 2016-03-20T19:05:10Z
swarm06: 10.42.0.148:2375
└ Status: Healthy
└ Containers: 11
└ Reserved CPUs: 0 / 8
└ Reserved Memory: 4 GiB / 16.4 GiB
└ Labels: executiondriver=native-0.2, kernelversion=4.4.4-std-3, operatingsystem=Ubuntu 14.04.4 LTS, storagedriver=btrfs
└ Error: (none)
└ UpdatedAt: 2016-03-20T19:04:40Z
Plugins:
Volume:
Network:
Kernel Version: 4.4.4-std-3
Operating System: linux
Architecture: amd64
CPUs: 24
Total Memory: 82.22 GiB
Name: swarm04

docker version

Client:
Version: 1.10.3
API version: 1.22
Go version: go1.5.3
Git commit: 20f81dd
Built: Thu Mar 10 15:54:52 2016
OS/Arch: linux/amd64

Server:
Version: swarm/1.1.3
API version: 1.22
Go version: go1.5.3
Git commit: 7e9c6bd
Built: Wed Mar 2 00:15:12 UTC 2016
OS/Arch: linux/amd64

larsla · 2016-03-28T13:02:33Z

I keep having this issue. Now even restarting the docker daemon doesn't help.
Please let me know what more information you need in order to find a solution for this.

mavenugo · 2016-03-28T15:24:25Z

@larsla invalid argument typically suggests its a kernel specific issue. I see that you are using Kernel Version: 4.4.4-std-3. Most likely its to do with that. Please share more details on your environment.

mavenugo · 2016-03-28T15:25:34Z

For those hitting the file exists problem, it would be of great help if someone can share some reproduction steps.

- Fixes moby#985 - Fixes moby#945 - Log time taken to set sandbox key - Limit number of concurrent DNS queries Signed-off-by: Madhu Venugopal <madhu@docker.com>

mavenugo · 2016-03-31T15:49:07Z

Closed via #1065 and is vendored into docker/docker.

Can someone try the docker/docker master and confirm the fix ?

- Fixes moby/libnetwork#1051 - Fixes moby/libnetwork#985 - Fixes moby/libnetwork#945 - Log time taken to set sandbox key - Limit number of concurrent DNS queries Signed-off-by: Madhu Venugopal <madhu@docker.com> (cherry picked from commit 90bb530)

brettdh · 2016-04-05T18:04:29Z

@mavenugo Unfortunately, once the problem has manifested, it's too late. I saw this today after upgrading my swarm hosts to docker 1.10.3, and if I'm reading the PR correctly, it prevents the problem from occurring, but it won't help for me to upgrade docker and try again. So I ended up rebooting the hosts again.

I greatly look forward to the release of 1.11.0, though, so this can hopefully be in the past. :-)

gittycat · 2016-04-05T22:46:41Z

@brettdh I noticed the same. Only a reboot of the host fixes it but this is not an option for production. The process can take hours in an enterprise IT environment where the vendor (us) have no direct control over the hypervisor.

matevarga · 2016-04-21T18:29:39Z

Same here.

Containers: 55
 Running: 0
 Paused: 0
 Stopped: 55
Images: 55
Server Version: 1.11.0
Storage Driver: aufs
 Root Dir: /var/lib/docker/aufs
 Backing Filesystem: extfs
 Dirs: 282
 Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: overlay bridge null host
Kernel Version: 4.2.0-34-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 7.792 GiB
Name: xxxxx
ID: 2R52:ZXPR:UD56:KTME:JNOW:2X76:Y6LM:3UCU:4PH6:7ONV:PT6X:ULIL
Docker Root Dir: /var/lib/docker
Debug mode (client): false
Debug mode (server): false
Username: mvarga
Registry: https://index.docker.io/v1/
Cluster store: consul://xxxx:8500
Cluster advertise: xxxx:2376

brettdh · 2016-05-16T17:15:02Z

This is still occurring on 1.11.1. (The file exists problem, specifically)

brettdh · 2016-06-16T14:28:51Z

And once again today.

ping @mavenugo

sebi-hgdata · 2016-06-23T13:55:40Z

@thaJeztah @mavenugo This should be reopened as it can be reproduced using the below script.
The idea is to simulate ungraceful docker daemon shutdowns when there are containers connected to an overlay network having a restart policy.

#!/usr/bin/env groovy

Thread.start {
    watchOutput("error")
}
deleteContainers()
c = "sudo docker network create -d overlay test".execute().text
print c

for (int i = 0; i < 100; i++) {
    def out = "sudo docker run -td --restart unless-stopped --net test --name t$i ubuntu:14.04.2 bash".execute().text
    print out
}
def resp = 'sudo docker ps'.execute().pipeTo("wc -l".execute()).text.trim()
assert resp == "101"

while (true) {
    println 'killing the daemon'
    killDockerDaemon()
    sleep(5000)
    resp = 'sudo docker ps'.execute().pipeTo("wc -l".execute()).text.trim()
    assert resp == "101"
}

//----------

def killDockerDaemon() {
    def c = ['bash', '-c', "sudo kill -9 \$(sudo service docker status|cut -d ' ' -f4)"].execute().text
    println c
}

def deleteContainers() {
    def c = ['bash', '-c', "sudo docker rm -f \$(sudo docker ps -a -q )"].execute().text
    println c
}

def watchOutput(text, file = "/var/log/upstart/docker.log") {
    Process p = "sudo tail -f $file".execute().pipeTo(["grep", "$text"].execute())
    p.in.eachLine { line ->
        println line
    }
}

My env is :

$ sudo docker info
Containers: 100
 Running: 0
 Paused: 0
 Stopped: 100
Images: 377
Server Version: 1.11.2
Storage Driver: overlay
 Backing Filesystem: extfs
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins: 
 Volume: local
 Network: overlay bridge null host
Kernel Version: 4.3.0-040300-generic
Operating System: Ubuntu 14.04.4 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 15.16 GiB
Name: seb
ID: 3IIK:AWIX:PLOR:BPQ4:XNEL:SSXQ:2GUL:VEKX:OVCQ:SCCX:MN2U:DTWH
Docker Root Dir: /home/seb/hgdata/deployments/docker
Debug mode (client): false
Debug mode (server): false
Registry: https://index.docker.io/v1/
WARNING: No swap limit support
Cluster store: consul://localhost:8500
Cluster advertise: 192.168.123.209:2375

$ sudo docker version
Client:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

Server:
 Version:      1.11.2
 API version:  1.23
 Go version:   go1.5.4
 Git commit:   b9f10c9
 Built:        Wed Jun  1 21:47:50 2016
 OS/Arch:      linux/amd64

mrjana · 2016-06-23T17:23:24Z

@sebi-hgdata Can you try this same script with 1.12-rc2 because we have added fixes in it which should fix it. If it is still present then atleast there will be some opportunity to fix it completely before the 1.12 release is out.

sebi-hgdata · 2016-06-23T20:19:02Z

@mrjana seems like the issue does not reproduce with 1.12-rc2 all containers restart properly during testing ... but I see a lot of errors like

time="2016-06-23T23:06:54.858306166+03:00" level=error msg="Failed to start container c0ecaca06cb75c2ff5e0950f5554053b593752f946201a5104c2e0d6c07ba44d: could not add veth pair inside the network sandbox: error setting interface \"vethc1dc8a1\" master to \"br\": Device does not exist"

after letting the script kill the daemon a couple of times, killing the script, stopping the docker daemon and starting it up again... after this no container is restarted because they failed with the above error.
I can create another issue if it's necessary

mrjana · 2016-06-23T20:24:16Z

@sebi-hgdata Thanks for testing. Yes please create another issue for the new bug you are seeing. We will make sure it is fixed before 1.12 release.

sebi-hgdata · 2016-06-23T20:39:46Z

@mrjana added #1290

bvis · 2016-11-04T15:24:51Z

I've just experienced it in a Swarm cluster running 1.12.3.

subnet sandbox join failed for \"10.0.2.0/24\": error creating vxlan interface: file exists

It has been produced when I did:

docker service rm logspout
docker service create --mode global --name logspout ...

schmunk42 · 2016-11-04T20:55:39Z

Still seeing this on swarm 1.2.5 + Docker 12.2-rc1

marcosnils · 2016-11-21T18:58:45Z

@mavenugo we're also seeing this quite often in PWD (play-with-docker) using 1.13rc1

hamburml · 2016-11-29T14:44:54Z

Had the file exists error with

Client:
 Version:      1.13.0-rc2
 API version:  1.25
 Go version:   go1.7.3
 Git commit:   1f9b3ef
 Built:        Wed Nov 23 06:17:45 2016
 OS/Arch:      linux/amd64

Server:
 Version:             1.13.0-rc2
 API version:         1.25
 Minimum API version: 1.12
 Go version:          go1.7.3
 Git commit:          1f9b3ef
 Built:               Wed Nov 23 06:17:45 2016
 OS/Arch:             linux/amd64
 Experimental:        false

with docker engine in swarm mode. After restarting docker service everything worked.

marcosnils · 2016-11-29T14:47:33Z

@Michael-Hamburger / all this will be fixed by #1574

I've applied this patch manually and haven't had any issues so far.

kinghuang · 2016-11-30T16:56:36Z

I'm running into this problem on my 1.12.3 swarm. Is there a workaround for when it happens? I've tried removing and adding the problematic network(s) without success. Is there something short of a full restart of the swarm hosts that can alleviate this?

marcosnils · 2016-11-30T18:05:53Z

Is there something short of a full restart of the swarm hosts that can alleviate this?

Yes, if you restart the daemon the problem should be gone. @mavenugo. But if you keep removing / creating networks you'll come across with it again

alexanderkjeldaas · 2017-05-10T19:32:25Z

I saw this now.

Docker version 17.05.0-ce, build 89658be

Restarting the docker daemon is not fixing this.

"starting container failed: subnet sandbox join failed for "10.0.2.0/24": error creating vxlan interface: file exists"

thaJeztah · 2017-05-11T08:29:05Z

@alexanderkjeldaas can you open a new issue with details, possibly it's different from the one that's resolved by #1574

discotroy · 2017-05-18T20:35:11Z

I took the liberty of opening a new thread as I, too, am encountering this issue on 17.05-ce (and have on earlier releases).

See #1765 for info.

roboticsexpert · 2018-06-26T19:42:19Z

@arteal
we rebooted our machine after that machine not booted 😆

krearthur · 2018-10-26T11:00:40Z

Same issue:
subnet sandbox join failed for "10.255.0.0/16": error creating vxl...

promzeus · 2019-06-12T13:21:12Z

a similar problem is reproduced now june 2019

docker info

Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 18.09.6
Storage Driver: zfs
Zpool: rpool
Zpool Health: ONLINE
Parent Dataset: rpool/ROOT/pve-1
Space Used By Parent: 1047266021376
Space Available: 227269107712
Parent Quota: no
Compression: on
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: active
NodeID: oo0dscwkoviwtnvocttkg9932
Is Manager: true
ClusterID: 3gfm66fjmcvn6q9k2mssqwcip
Managers: 1
Nodes: 1
Default Address Pool: 10.0.0.0/8
SubnetSize: 24
Orchestration:
Task History Retention Limit: 5
Raft:
Snapshot Interval: 10000
Number of Old Snapshots to Retain: 0
Heartbeat Tick: 1
Election Tick: 10
Dispatcher:
Heartbeat Period: 5 seconds
CA Configuration:
Expiry Duration: 3 months
Force Rotate: 0
Autolock Managers: false
Root Rotation In Progress: false
Node Address: 172.16.1.49
Manager Addresses:
172.16.1.49:2377
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.15.18-9-pve
Operating System: Debian GNU/Linux 9 (stretch)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.31GiB
Name: centroit-pve1
ID: 6INQ:DFKN:3V2I:XBIZ:VDMR:7HJH:VKEY:OMNY:7YIT:YFK7:KA74:MZLO
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

# iptables-save
# Generated by iptables-save v1.6.0 on Wed Jun 12 16:21:32 2019
*mangle
:PREROUTING ACCEPT [525538:905157812]
:INPUT ACCEPT [492687:891964017]
:FORWARD ACCEPT [162666:115681384]
:OUTPUT ACCEPT [312922:803390219]
:POSTROUTING ACCEPT [475588:919071603]
COMMIT
# Completed on Wed Jun 12 16:21:32 2019
# Generated by iptables-save v1.6.0 on Wed Jun 12 16:21:32 2019
*nat
:PREROUTING ACCEPT [1659:93104]
:INPUT ACCEPT [393:22550]
:OUTPUT ACCEPT [3862:570483]
:POSTROUTING ACCEPT [5128:641037]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A DOCKER -i docker0 -j RETURN
COMMIT
# Completed on Wed Jun 12 16:21:32 2019
# Generated by iptables-save v1.6.0 on Wed Jun 12 16:21:32 2019
*filter
:INPUT ACCEPT [326852:574844648]
:FORWARD ACCEPT [104813:74490157]
:OUTPUT ACCEPT [209763:530418909]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Wed Jun 12 16:21:32 2019

ProteanCode · 2020-01-16T02:03:12Z

Still happening on docker swarm at ubuntu 18.04
Solution that worked for me was to remove stack and redeploy it

docker stack rm jenkins
docker stack deploy -c docker-compose.yml jenkins

 Debug Mode: false

Server:
 Containers: 50
  Running: 30
  Paused: 0
  Stopped: 20
 Images: 114
 Server Version: 19.03.5
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: active
  NodeID: sywquid85sgsma3puf4gw6u68
  Is Manager: true
  ClusterID: zh6x2htbsuevss97ytz2kbkn5
  Managers: 1
  Nodes: 1
  Default Address Pool: 10.0.0.0/8  
  SubnetSize: 24
  Data Path Port: 4789
  Orchestration:
   Task History Retention Limit: 5
  Raft:
   Snapshot Interval: 10000
   Number of Old Snapshots to Retain: 0
   Heartbeat Tick: 1
   Election Tick: 10
  Dispatcher:
   Heartbeat Period: 5 seconds
  CA Configuration:
   Expiry Duration: 3 months
   Force Rotate: 0
  Autolock Managers: true
  Root Rotation In Progress: false
  Node Address: 51.77.42.145
  Manager Addresses:
   51.77.42.145:2377
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.15.0-74-generic
 Operating System: Ubuntu 18.04.3 LTS
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 31.2GiB
 Name: ns3145583
 ID: O3MT:IN6V:IFUN:MMQ4:77FH:H7A2:CUUP:3ZIU:3FSS:JKBW:JADU:SKQ3
 Docker Root Dir: /home/dockerd
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support```

bodhi-one · 2022-03-10T22:51:36Z

This error still occurs occasionally in Docker 20.x, VM restart typically required.

brettdh mentioned this issue Mar 3, 2016

Docker engine 1.9.0-rc1 does not gracefully restart moby/moby#17083

Closed

mavenugo added a commit to mavenugo/libnetwork that referenced this issue Mar 31, 2016

changelog for 0.7.0-rc.1

0ab34e9

- Fixes moby#985 - Fixes moby#945 - Log time taken to set sandbox key - Limit number of concurrent DNS queries Signed-off-by: Madhu Venugopal <madhu@docker.com>

mavenugo mentioned this issue Mar 31, 2016

changelog for 0.7.0-rc.1 #1066

Merged

mavenugo mentioned this issue Mar 31, 2016

Vendor Libnetwork v0.7.0-rc.1 moby/moby#21671

Merged

kencochrane closed this as completed in kencochrane/docker@90bb530 Mar 31, 2016

sebi-hgdata mentioned this issue Jun 23, 2016

Failed to start container xxxx: could not add veth pair inside the network sandbox: error setting interface \"vethc1dc8a1\" master to \"br\": Device does not exist #1290

Closed

schmunk42 mentioned this issue Nov 4, 2016

Containers on overlay network and "error creating vxlan interface: file exists" moby/moby#21482

Closed

discotroy mentioned this issue May 18, 2017

Error creating vxlan interface: file exists #1765

Open

thaJeztah mentioned this issue Jul 16, 2019

Use os.ReadDir instead of filepath.Walk in netns cleanup moby/moby#36511

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945

Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945

brettdh commented Feb 11, 2016

tomashejatko commented Feb 25, 2016

brettdh commented Mar 2, 2016

tomashejatko commented Mar 2, 2016

frntn commented Mar 16, 2016

larsla commented Mar 20, 2016

larsla commented Mar 28, 2016

mavenugo commented Mar 28, 2016

mavenugo commented Mar 28, 2016

mavenugo commented Mar 31, 2016

brettdh commented Apr 5, 2016

gittycat commented Apr 5, 2016

matevarga commented Apr 21, 2016 •

edited

brettdh commented May 16, 2016 •

edited

brettdh commented Jun 16, 2016

sebi-hgdata commented Jun 23, 2016 •

edited

mrjana commented Jun 23, 2016

sebi-hgdata commented Jun 23, 2016 •

edited

mrjana commented Jun 23, 2016

sebi-hgdata commented Jun 23, 2016

bvis commented Nov 4, 2016 •

edited

schmunk42 commented Nov 4, 2016

marcosnils commented Nov 21, 2016 •

edited

hamburml commented Nov 29, 2016

marcosnils commented Nov 29, 2016

kinghuang commented Nov 30, 2016

marcosnils commented Nov 30, 2016 •

edited

alexanderkjeldaas commented May 10, 2017 •

edited

thaJeztah commented May 11, 2017

discotroy commented May 18, 2017

roboticsexpert commented Jun 26, 2018

krearthur commented Oct 26, 2018

promzeus commented Jun 12, 2019 •

edited

ProteanCode commented Jan 16, 2020 •

edited

bodhi-one commented Mar 10, 2022

Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945

Cannot start container: subnet sandbox join failed for "10.0.0.0/24": error creating vxlan interface: file exists #945

Comments

brettdh commented Feb 11, 2016

tomashejatko commented Feb 25, 2016

brettdh commented Mar 2, 2016

tomashejatko commented Mar 2, 2016

frntn commented Mar 16, 2016

larsla commented Mar 20, 2016

docker info

docker version

larsla commented Mar 28, 2016

mavenugo commented Mar 28, 2016

mavenugo commented Mar 28, 2016

mavenugo commented Mar 31, 2016

brettdh commented Apr 5, 2016

gittycat commented Apr 5, 2016

matevarga commented Apr 21, 2016 • edited

brettdh commented May 16, 2016 • edited

brettdh commented Jun 16, 2016

sebi-hgdata commented Jun 23, 2016 • edited

mrjana commented Jun 23, 2016

sebi-hgdata commented Jun 23, 2016 • edited

mrjana commented Jun 23, 2016

sebi-hgdata commented Jun 23, 2016

bvis commented Nov 4, 2016 • edited

schmunk42 commented Nov 4, 2016

marcosnils commented Nov 21, 2016 • edited

hamburml commented Nov 29, 2016

marcosnils commented Nov 29, 2016

kinghuang commented Nov 30, 2016

marcosnils commented Nov 30, 2016 • edited

alexanderkjeldaas commented May 10, 2017 • edited

thaJeztah commented May 11, 2017

discotroy commented May 18, 2017

roboticsexpert commented Jun 26, 2018

krearthur commented Oct 26, 2018

promzeus commented Jun 12, 2019 • edited

docker info

ProteanCode commented Jan 16, 2020 • edited

bodhi-one commented Mar 10, 2022

matevarga commented Apr 21, 2016 •

edited

brettdh commented May 16, 2016 •

edited

sebi-hgdata commented Jun 23, 2016 •

edited

sebi-hgdata commented Jun 23, 2016 •

edited

bvis commented Nov 4, 2016 •

edited

marcosnils commented Nov 21, 2016 •

edited

marcosnils commented Nov 30, 2016 •

edited

alexanderkjeldaas commented May 10, 2017 •

edited

promzeus commented Jun 12, 2019 •

edited

ProteanCode commented Jan 16, 2020 •

edited