Failed to start container in overlay network since "br0" exchange full

[BSWANG]

I am using docker overlay network, my docker-engine information on below.

$ docker info
Containers: 5
Images: 32
Server Version: 1.9.1
Storage Driver: overlay
 Backing Filesystem: extfs
Execution Driver: native-0.2
Logging Driver: json-file
Kernel Version: 4.3.0-1.el7.elrepo.x86_64
Operating System: CentOS Linux 7 (Core)
CPUs: 2
Total Memory: 3.858 GiB
Name: cc821b04098444cb0a020171b6bbc2630-node1
ID: OZI5:2WWI:5GFY:HV5V:YWWS:DDLU:ASPW:D6F2:572T:LYJB:6R5F:BBW6
Cluster store: etcd://10.201.145.128:2379/cc821b04098444cb0a020171b6bbc2630
Cluster advertise: 10.165.126.76:2376

When I create and delete containers in overlay network with at least one container in the network. I will get could not add veth pair inside the network sandbox: error setting interface "vethe1e64f7" master to "br0": exchange full error, and the container remain on created state.

It seems docker daemon not delete the veth pair when deleting containers, and get bridge full when continue create containers in the network since cannot create veth on sandbox's default bridge "br0".

There is one test case below:

#create docker overlay network
docker network create -d overlay multi-host-network
#ensure at least one container in the overlay network
docker run -itd --net multi-host-network busybox
#create and remove container in the network
for (( i=0; i<1024; i++ )) do docker run -itd --net multi-host-network --name test$i busybox; docker rm -f test$i; done;

And I get

...
d7064a4a6218d34da6e4502294c325184780e603c802643bbb666aa44c579347
test1020
2dcabbab16238a34739e850e48e99e32eac2f5c504cf0207afbc475dfa9ebf26
test1021
afa928d3684e7d35d0ce9898262b45148dbb53d52fe60e5041c8a0b07b4a8250
Error response from daemon: Cannot start container afa928d3684e7d35d0ce9898262b45148dbb53d52fe60e5041c8a0b07b4a8250: could not add veth pair inside the network sandbox: error setting interface "veth5cbfa6a" master to "br0": exchange full
test1022
...

[wildsurfer]

is there any workaround to fix this on live environment?

[wildsurfer]

My workaround was to stop all containers, remove them, remove network overlay, create it with the same name and start all containers back:

$ docker-compose stop
$ docker-compose rm
$ docker network rm mynetwork
$ docker network create mynetwork
$ docker-compose up -d

[BSWANG]

@wildsurfer remove all containers in the overlay network can fix this, or just restart docker daemon can work too.