Discovery temporarily loses nodes

[aluzzardi]

Now that we have node removal, it's easier to spot those issues.

I was using consul discovery and noticed this:

INFO[0000] Registered Engine dev at 127.0.0.1:4242
INFO[0035] Removed Engine dev
INFO[0055] Registered Engine dev at 127.0.0.1:4242

It means that the node was lost after 35 minutes and re-discovered again 20 minutes later, even though nothing changed (both the manager and join were still running).

[abronan]

What were your ttl, heartbeat and refresh (on Manager) settings during this test? Might be related to #830 although not sure.

[abronan]

@aluzzardi I have a running Swarm with Consul and default settings (and multiple managers) for more than 2 hours now, I cannot reproduce :( Are you sure this was not something related to your VMs setup, or putting your machine into sleep while testing?

[abronan]

Seems like the node removal was a legitimate one if you were using default values, not sure if this is the Hub being down (for token discovery) or another network related error if you were using another discovery service. I'll close it for now.