Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Unable to remove a node from the cluster #197

Closed
mountkin opened this Issue Dec 22, 2014 · 16 comments

Comments

Projects
None yet
7 participants
@mountkin
Copy link
Contributor

commented Dec 22, 2014

The Watch method of the discovery backend calls cluster.UpdateNodes method to update the node list according the return value of the Fetch method of the discovery service.
But it only inserts the nodes that aren't exist in the node list and doesn't touch the nodes that are "removed" from the discovery service.

If a node is removed from the discovery service, the swarm list command returns the correct result because it calls the Fetch method of the discovery service directly. But in fact the removed node is still in the node list of the cluster.

@vieux vieux added the kind/bug label Dec 22, 2014

@vieux

This comment has been minimized.

Copy link
Contributor

commented Dec 22, 2014

That's correct, right now, the node isn't removed, but flagged as unhealthy and is skipped by the scheduler.

We are not sure about how we want to handle this, because we don't want to loose information in case of the node comes back up.

Did you encounter a specific issue ?

@mountkin

This comment has been minimized.

Copy link
Contributor Author

commented Dec 23, 2014

Hi, @vieux
As far as I can see the removed node is not flagged as unhealthy unless the refreshContainers method returns an error.
My case is that after I add a node to the cluster and then need to remove it for debugging, I don't want new containers to be launched on the removed node.

@mountkin

This comment has been minimized.

Copy link
Contributor Author

commented Dec 23, 2014

What about add another command such as swarm leave --addr=1.2.3.4 or swarm remove --addr=1.2.3.4 that sends a "leave" message to the discovery service? The swarm daemon can remove the corresponding node when it receives the message.
Or perhaps we can send a specific signal to the swarm join process, when the process receives the signal it send the "leave" message to the discovery service and quit.

For tracking the information of the removed node when it comes back up, perhaps a persistent storage of the nodes information is required.

@vieux

This comment has been minimized.

Copy link
Contributor

commented Dec 23, 2014

@mountkin I don't feel live a new command would be the solution. If you're using etcd or consul, there is a TTL so your node will disappear. This feature is coming to the hosted registry service.

@vieux

This comment has been minimized.

Copy link
Contributor

commented Dec 23, 2014

@aluzzardi what do you think ?

@aluzzardi

This comment has been minimized.

Copy link
Contributor

commented Dec 29, 2014

Eventually, we will need to handle the node removal case correctly.

Right now we don't do any house keeping even if the node gets removed through etcd / consul.

We need to find a way to make the distinction between node removal and unhealthy node.

Regarding the debugging, I don't think nodes should be removed straight from discovery as every implementation will act differently. Perhaps this should be a super set of the API (e.g. DELETE /v1/nodes/foo) and the swarm binary would provide easy access to it.

@tnachen

This comment has been minimized.

Copy link
Contributor

commented Dec 31, 2014

Different schedulers also will have their own health check and tracking of different task status, such as Mesos.
I don't have a good picture how different schedulers are suppose to plug into swarm yet, but I think ideally is that there is both a way to notify swarm for actually node health, or let swarm's node detection and tracking to be pluggable so it can call out to existing services. In Mesos case it can simply contact the master to get all the information.

@vincentwoo

This comment has been minimized.

Copy link

commented Jan 4, 2015

👍 for node removal being a priority. If you have long running jobs, you need some way of signaling that a node is going to be removed but is still present while it is finishing its outstanding jobs.

@bryantidd

This comment has been minimized.

Copy link

commented Jan 27, 2015

what is the priority and timeline on node removal - housekeeping? It seems that this is integral to using Swarm. Otherwise it presents only a half solution for Docker cluster management. Would be happy to help test.

@mountkin

This comment has been minimized.

Copy link
Contributor Author

commented Jan 28, 2015

The node list is synced from discovery services periodically.
If we simply remove it from swarm's cached node list, it'll be added again when the next UpdateNodes is triggered by discovery service.

I think the problem can be solved in two ways:

  1. Add a new member to the Cluster struct to keep all the nodes that had been "removed". And modify the UpdateNodes function to exclude the nodes that are in the removed list.
    The list of removed nodes should be stored persistently.
    This will introduce two new APIs to swarm:
    • DELETE /nodes/:node_id which is responsible for add an entry to the removed list and delete the node from cluster.nodes.
    • POST /nodes/:node_id which is responsible for remove the node from the removed list.
  2. The user removed the node from discovery service first, then call swarm's API to delete it from the cache.
    But this may not work for the discovery services that haven't exposed a DELETE API, such as nodes and token.

@vieux @aluzzardi What do you think?

@vincentwoo

This comment has been minimized.

Copy link

commented Apr 2, 2015

@vieux saw from your comments in linked issues that this was going to be addressed soon - can you give us any updates on how this is coming together?

@aluzzardi

This comment has been minimized.

Copy link
Contributor

commented May 18, 2015

@vincentwoo We have a few PRs cooking for this - should be completed for 0.3.0.

@vincentwoo

This comment has been minimized.

Copy link

commented Oct 26, 2015

@aluzzardi I see that this was marked resolved in #811, but I don't see the topic addressed in the official docs or changelog. Can you describe the procedure required to remove a node from the cluster? Does it depend on the discovery service?

@abronan

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2015

Hi @vincentwoo, Yes it depends on the discovery service. If using the kv discovery or the token discovery, there is a TTL field attached to the node entry. If the node does not heartbeat to notify of its presence in the cluster (ie. stopping the swarm join or the node crashes), the TTL will expire and the Manager will be notified to remove the node entry.

@vincentwoo

This comment has been minimized.

Copy link

commented Oct 26, 2015

So if I understand you correctly, the way to remove a node is to explicitly kill whatever background process swarm join is running in, and wait for the TTL to expire? Are there plans to encapsulate or document this?

@abronan

This comment has been minimized.

Copy link
Contributor

commented Oct 26, 2015

Yes exactly. The documentation mentions the heartbeat flag but I agree that this should be explained better.. Especially about the whole process off tagging a node as dead to evict it from any scheduling decision and then removing it from the cluster, the process of heartbeating and watch notifications, etc.

You can maybe open an issue to track this. I'll look into enhancing the docs for this aspect.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.