Failed to send gossip

[polonskiy]

When I try to run ~40 containers with serf I get this:

2015/02/10 17:00:38 [ERR] memberlist: Failed to send ack: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send ping: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.60:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.52:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.90:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.82:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.76:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.60:7946: write udp: invalid argument
2015/02/10 17:00:39 [ERR] memberlist: Failed to send gossip to 172.17.4.89:7946: write udp: invalid argument
2015/02/10 17:00:40 [ERR] memberlist: Failed to send gossip to 172.17.4.89:7946: write udp: invalid argument
2015/02/10 17:00:40 [ERR] memberlist: Failed to send gossip to 172.17.4.81:7946: write udp: invalid argument

Docker version 1.4.1, build 5bc2ff8
Serf v0.6.3
Agent Protocol: 4 (Understands back to: 2)
Ubuntu 14.04.1 LTS

Example:

serf_cid=$(docker run -d -v $PWD/serf:/usr/local/bin/serf ubuntu serf agent)
serf_ip=$(docker inspect --format '{{.NetworkSettings.IPAddress}}' $serf_cid)
for i in {1..40}; do docker run -d -v $PWD/serf:/usr/local/bin/serf ubuntu serf agent -join $serf_ip; done
docker logs $serf_cid

[ryanuber]

@polonskiy we have seen odd UDP routing issues while running inside of Docker containers a number of times. It has something to do with the ARP cache, though I don't remember specifically. We've also seen cases where conntrack interferes with UDP, so you might want to try flushing if you have that enabled.

[ryanuber]

@polonskiy it may also help to add a sleep between the join's. These UDP problems usually manifest during rapid setup/teardown.

[polonskiy]

@ryanuber yes, that was a problem with ARP table overflow (as far as I understood). The solution is to set bigger net.ipv*.neigh.default.gc_thresh* values on the host system. https://gist.github.com/polonskiy/00a71bab32360ffcb79f
Thanks.

[ryanuber]

@polonskiy I am going to close this, as it sounds like the issue is resolved. If you are still having problems, please comment and we can re-open this. Thanks!