Serf health check failed randomly

[jiangshengwu]

I have made a three-server consul cluster which is well-functioning on both service discovery and dns forwarding.

What confuses me is that, on the web-ui, one or two of the three consul server nodes become yellow randomly and occasionally, which represents its failure of serf health check.

And there're a lot of logs like this below:

2015/06/12 08:04:26 [INFO] memberlist: Marking consul-server3 as failed, suspect timeout reached
2015/06/12 08:04:26 [INFO] serf: EventMemberFailed: consul-server3 10.170.201.26
2015/06/12 08:04:26 [INFO] consul: removing server consul-server3 (Addr: 10.170.201.26:8300) (DC: dc1)
2015/06/12 08:04:26 [INFO] consul: member 'consul-server3' failed, marking health critical

I'm not sure why it behaves this way.

I start the server nodes with:

docker run -itd --name consul-server1 -h consul-server1 \
    -p 8300:8300 -p 8301:8301 -p 8301:8301/udp -p 8302:8302 \
    -p 8302:8302/udp -p 8400:8400 -p 8500:8500 -p 10.173.5.20:53:53 \
    -p 10.173.5.20:53:53/udp progrium/consul -server \
    -bootstrap -log-level debug -advertise 10.173.5.20

consul-server2 and consul-server3 are almost the same as that above, except for the ip address and using -join instead of -bootstrap.

Where have I done wrong?

[ryanuber]

Hi @jiangshengwu, this is normally a sign of UDP routing issues. Docker has a well-known ARP cache issue that is discussed at length in #352.

You might try some of the solutions suggested there to see if it helps.

[jiangshengwu]

@ryanuber Thanks for your reply. I've read the issue you referred to. It helps a lot.

So I'll just close this one.