-
Notifications
You must be signed in to change notification settings - Fork 287
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
ARP Cache purging docoumentation #26
Comments
I'm experiencing the same issue. Running Ubuntu on EC2 and nothing other than waiting seems to do the trick. Some pointers on what you found that worked would be great. |
the documentation refer to gratuitous ARP reply sent to the local network segment ( of consul ). That would allow all neighbour hosts to update there ARP table with the new MAC, restoring connectivity. Try running that from the container ( through docker exec, or as per OP using nsenter ). Also, it seem this ARP cache issue had been addressed directly upstream in docker : |
FWIW, I just ran into this in our deployed system and unfortunately the arping didn't do the trick. Of course there could be other factors, but just thought I'd let folks know. |
So...what's the right way to work around this? |
I can't seem to make any of the above workarounds work. The only thing that works for me is to shut the container down for 5 minutes and then start it again. Docker: 1.5.0 |
@johnrengelman. wondering, can your try to hard code the mac of the container with --mac-address or maybe using --net=host in your docker run for consul ? |
I'll give those a try hopefully today...off on something else at the moment. |
@johnrengelman were you able to try suggested mitigations? Seems like we've been running into this issue, initially opened a ticket on consul: hashicorp/consul#738 What I really need to know is if there is any valid workaround or if this issue has been addressed in a newer version of docker (we're running 1.4.1). This has been a persistent problem for us, making it difficult to try and use our consul cluster for some more interesting applications, where uptime is more critical. In the end we're probably going to be forced to go another route. |
I am still seeing this, even with docker 1.5. Using --net=host, while not ideal, does appear to sidestep this issue. |
This works for me as a temprorary workaround: just remove The (all) containers with: then rebuild and run. |
I spent a large chunk of the day digging into this. I flushed every ARP cache from here to Timbuktu with no improvement. However, this worked for me: hashicorp/consul#352 (comment) Try |
If anyone else is on CoreOS like me, you can use my Docker image to do this:
|
upvote. This thing is really annoying. conntrack -F not helping at all on ubuntu14lts hosts. |
👍 conntrack -F doesn't work for me also on ubuntu 12.04 LTS... |
+1 had the same issue running centos. Ended up dropping docker-consul for now and install things the old way. Happy to help fixing the issue although *nix networks is not my cup of tea. |
I'm on docker 1.7. |
I'm on docker 1.9.0 with CentOS 7, running consul with |
I'd like to point out the best working solution I could find was removing the docker completely Kill all containerssudo docker kill $(sudo docker ps -a -q) Delete all containerssudo docker rm $(sudo docker ps -a -q) Delete all imagessudo docker rmi $(sudo docker images -q) Then I run then bring the docker back up. Hope this helps |
Can someone explain what the actionable item is for this issue (is it still an issue?) and give an updated way to reproduce the issue ( |
Removing the data dir on host worked for me... it also deleted all my vault secrets 😂 Which I think might be why @twhart might've got it working. |
This is still relevant. The best (and about the only working) workaround so far is to manually clear the UDP cache of I've added the following to my deployment script after tearing down the old Consul server container and before starting the new container:
|
I've faced this issue running a small cluster of ambari/hdp services. Ambari has using consul as nameserver to coordinate the members of the cluster. And even cluster members has shutted down, they can't be startup correctly due consul neither have other ip or is unable to register new members. |
in the README.me it mentions "You can also manually reset the cache." -> If there is a reproducible solution, can you please supply the exact command line that will perform this? Is this something that will need to only be run on the host that went through a consul restart, or any other hosts in the cluster? does it need to be run in the docker container networking namespace (e.g using nsenter --net) or simply on the host?
I've been running into this issue and trying the following things on the host doesn't seem to help (either after stopping docker-consul and/or the docker daemon, or while either one of them is still running). only thing that works is waiting a few minutes, but thats not an acceptable solution for my environment atm.
The text was updated successfully, but these errors were encountered: