New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Proposal: Container Rescheduling #1488

Closed
aluzzardi opened this Issue Dec 3, 2015 · 39 comments

Comments

Projects
None yet
@aluzzardi
Contributor

aluzzardi commented Dec 3, 2015

Background

The goal of this proposal is to reschedule containers automatically in case of node failure.

This is currently one of the top requested feature for Swarm.

Configuration

The behavior should be user controllable and disabled by default since rescheduling can have nasty effects on stateful containers.

The user can select the policy at run time use the reschedule environment variable:

docker run -e reschedule:on-node-failure redis

Possible values for reschedule are:

  • no: Never reschedule the container (default).
  • on-node-failure: Reschedule the container whenever the node fails.

The reason this is more complicated than yes/no is in the future we might have more complicated rescheduling policies (for instance, we might want reschedule containers to re-spread or re-pack them). Open question: Is this really necessary?

Rescheduling policies will be stored as a container label: com.docker.swarm.reschedule-policy

Persistance

Ideally, Swarm would store all containers (at least those that should be rescheduled) persistently. That way, the manager can figure out which containers are down and take action.

Unfortunately, we currently don't have a shared state and this feature has been postponed because of that for a long time.

Since this is one of the top requested feature, I propose we take a different approach until we have shared state (that feature has been postponed for usability concerns - we don't want to make a kv store a dependency for Swarm).

By storing the rescheduling policy as a container label, we are able to reconstruct the desired state at startup time.

Since we are already storing constraints, affinities etc as container labels (exactly for this reason), the manager will have all the information it needs to perform rescheduling.

This means we can restart the manager as much as we want and it will resume rescheduling as expected.

However, the problem arises when a node goes down while the manager is not running: in that case, we won't "remember" that container even existed when the manager is started again.

This situation can be counter-balanced by using replication. The rescheduler would be running on the primary manager and, upon failure, the replica that gets elected primary would be taking over rescheduling responsibilities.

Since every manager is aware of the cluster state (containers & rescheduling policy), it means that as long as at least one manager is still running we won't forget about containers.

Failure detection

This functionality is already provided by cluster/engine.

Engine actively heartbeats nodes in the cluster every X seconds. After Y failures, the node is marked as unhealthy.

Rescheduling can rely on the health status already available.

Resurrection

Eventually, a node may come back to life and re-join the cluster. If the node has containers that were rescheduled, we will end up with duplicates.

Swarm should monitor incoming nodes and, upon detecting a duplicate container, it should destroy the oldest one (keeping the most recently created container alive). This behavior could eventually be configurable by the user (keep oldest, keep newer, ...), although we may want to avoid providing that option until we see a valid use case.

If duplicate containers were started with a --restart option, there is going to be a small window during which both containers are running at the same time. This can be a serious problem if only one instance of that container is supposed to run at one point.

We could force all containers that have rescheduling enabled to never automatically restart. In that case, whenever a node joins, Swarm could decide to either start the containers of destroy them if they are duplicates.

However, there are many drawbacks to this approach:

  • Restart policies have to be handled by Swarm. This introduces high complexity since we'd have to re-implement things such as --restart=on-failure:5 which require maintaining lots of state
  • If the manager is down, containers won't start automatically. This is a serious issue since it could lead to outages. Up until now, if Swarm is down the engines continue to operate normally and this change would break that contract.
  • Swarm might miss events, leading to containers not getting properly restarted

Furthermore, It doesn't actually entirely solve the issue. If the node didn't actually die (e.g. it just froze for a while, there was a netsplit, networking temporarily dropped, ...) we will end up with duplicate containers running for a while anyway.

Given all the potential issues that might arise by handling the restart policy on the Swarm side and the fact that duplicate containers may end up running at the same time anyway, I suggest we do not interfere with --restart and document the fact that rescheduled containers may be running in parallel for a short time window.

Networking

When rescheduling containers, Swarm must handle multi-host networking properly.

The goal is for the new container to take over the previous one.

In an overlay network setup, this may involve:

  • Making sure the new container takes over the IP address of the old container
  • Ensuring service discovery works properly
  • Cutting the old container off the network before starting the new one. Even though we presume the node to be down, it might still be up and running. Disconnecting the old container would alleviate side effects of duplicate containers.
@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Dec 3, 2015

Contributor

/cc @docker/swarm-maintainers @mavenugo @mrjana @jpetazzo @dnephin

Contributor

aluzzardi commented Dec 3, 2015

/cc @docker/swarm-maintainers @mavenugo @mrjana @jpetazzo @dnephin

@chanwit

This comment has been minimized.

Show comment
Hide comment
@chanwit

chanwit Dec 3, 2015

Contributor

Me too chose this in the survey 👍

Contributor

chanwit commented Dec 3, 2015

Me too chose this in the survey 👍

@cultureulterior

This comment has been minimized.

Show comment
Hide comment
@cultureulterior

cultureulterior Dec 3, 2015

I will note that Amazon ECS (which I'm now using, because swarm did not deliver this feature in time), does rebalancing by disabling local container restarts. I also don't think having a KV store as a dependency as out of the question, as long as it is swappable

cultureulterior commented Dec 3, 2015

I will note that Amazon ECS (which I'm now using, because swarm did not deliver this feature in time), does rebalancing by disabling local container restarts. I also don't think having a KV store as a dependency as out of the question, as long as it is swappable

@mrjana

This comment has been minimized.

Show comment
Hide comment
@mrjana

mrjana Dec 3, 2015

@aluzzardi Is IP stability a hard requirement? As long as we remap the container name to a different IP, the service discovery part should handle mapping any new client requests to the new IP(and hence to the new container). Wouldn't that be enough?

mrjana commented Dec 3, 2015

@aluzzardi Is IP stability a hard requirement? As long as we remap the container name to a different IP, the service discovery part should handle mapping any new client requests to the new IP(and hence to the new container). Wouldn't that be enough?

@calind

This comment has been minimized.

Show comment
Hide comment
@calind

calind Dec 4, 2015

For me this is the missing piece for running swarm in production.

I think that it's ok to have this feature dependant on a KV store, as docker overlay network requires it and I don't see any point of running swarm without overlay network (ok, this can be swapped but the majority of implementations rely on a kv store).

calind commented Dec 4, 2015

For me this is the missing piece for running swarm in production.

I think that it's ok to have this feature dependant on a KV store, as docker overlay network requires it and I don't see any point of running swarm without overlay network (ok, this can be swapped but the majority of implementations rely on a kv store).

@vieux vieux self-assigned this Dec 9, 2015

@clintkitson

This comment has been minimized.

Show comment
Hide comment
@clintkitson

clintkitson Dec 10, 2015

I would add the volume driver portion in here as well. If there is a container that has external volumes attached and is requested on another host, then it should be the case that the same volumes are brought to the new host.

In the case of REX-Ray (rexray/rexray#190) it now has pre-emption built into most drivers. This means that the new requesting container runtime will cause a forceful mount which detaches it from any host that currently has it. The setting is currently a global setting at a driver level for us, but it would be an interesting addition to the volume plugins to allow a flag on mount that gets used by Swarm to tell it to pre-empt or force mount in the case of re-scheduling. Typically we wouldn't want to enable pre-emption since it is a safety feature to block mounting from multiple hosts or block detaching/attching unintentionally. cc @cpuguy83

Drivers that don't have a forceful mount option or pre-emption will cause the containers that get requested on a new host to error since their volume will not be able to be dismounted. The exception here depends on the storage platform. For example, EC2 and OpenStack disallows this by default. This makes sense for safety, as we want to make people be explicit about mounting a volume to multiple hosts or doing detach/attach operations.

clintkitson commented Dec 10, 2015

I would add the volume driver portion in here as well. If there is a container that has external volumes attached and is requested on another host, then it should be the case that the same volumes are brought to the new host.

In the case of REX-Ray (rexray/rexray#190) it now has pre-emption built into most drivers. This means that the new requesting container runtime will cause a forceful mount which detaches it from any host that currently has it. The setting is currently a global setting at a driver level for us, but it would be an interesting addition to the volume plugins to allow a flag on mount that gets used by Swarm to tell it to pre-empt or force mount in the case of re-scheduling. Typically we wouldn't want to enable pre-emption since it is a safety feature to block mounting from multiple hosts or block detaching/attching unintentionally. cc @cpuguy83

Drivers that don't have a forceful mount option or pre-emption will cause the containers that get requested on a new host to error since their volume will not be able to be dismounted. The exception here depends on the storage platform. For example, EC2 and OpenStack disallows this by default. This makes sense for safety, as we want to make people be explicit about mounting a volume to multiple hosts or doing detach/attach operations.

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Dec 11, 2015

Contributor

@clintonskitson thanks, I'll update the proposal to include some text about volumes

Contributor

vieux commented Dec 11, 2015

@clintonskitson thanks, I'll update the proposal to include some text about volumes

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 12, 2015

Contributor

How come you want to use an extra label?
Using the restart policies seems like it would be good enough for rescheduling.
restart=always -- always reschedule
restart=on-failure -- reschedule if the failure count is ok, node failure should not affect the failure count
restart=unless-stopped -- reschedule if the last state was not stopped

Also need to account for paused containers... I have a feeling that these should not be rescheduled ever.

Contributor

cpuguy83 commented Dec 12, 2015

How come you want to use an extra label?
Using the restart policies seems like it would be good enough for rescheduling.
restart=always -- always reschedule
restart=on-failure -- reschedule if the failure count is ok, node failure should not affect the failure count
restart=unless-stopped -- reschedule if the last state was not stopped

Also need to account for paused containers... I have a feeling that these should not be rescheduled ever.

@schmunk42

This comment has been minimized.

Show comment
Hide comment
@schmunk42

schmunk42 Dec 12, 2015

I'd have a question about this topic.

Currently the docker daemon handles restarting of containers. But isn't there a conflict between daemon and swarm manager when it comes to rescheduling?

The (already discussed) scenario I am referring to is: There's a node-failure, swarm master would reschedule containers to healthy nodes. Now the failed node gets healed...

It comes up and the daemon starts containers with restart policy, but they will be duplicated, since swarm manager already rescheduled them.

So, should restarting/rescheduling handled by exclusively either docker daemon or swarm-manager?

schmunk42 commented Dec 12, 2015

I'd have a question about this topic.

Currently the docker daemon handles restarting of containers. But isn't there a conflict between daemon and swarm manager when it comes to rescheduling?

The (already discussed) scenario I am referring to is: There's a node-failure, swarm master would reschedule containers to healthy nodes. Now the failed node gets healed...

It comes up and the daemon starts containers with restart policy, but they will be duplicated, since swarm manager already rescheduled them.

So, should restarting/rescheduling handled by exclusively either docker daemon or swarm-manager?

@abronan

This comment has been minimized.

Show comment
Hide comment
@abronan

abronan Dec 12, 2015

Contributor

@schmunk42 Agreed, I commented on that on the old proposal: #599 (comment)

We need to clean one or the other or the container will end up being duplicated.

Contributor

abronan commented Dec 12, 2015

@schmunk42 Agreed, I commented on that on the old proposal: #599 (comment)

We need to clean one or the other or the container will end up being duplicated.

@geovanisouza92

This comment has been minimized.

Show comment
Hide comment

geovanisouza92 commented Dec 13, 2015

+1

@clintkitson

This comment has been minimized.

Show comment
Hide comment
@clintkitson

clintkitson Dec 13, 2015

I wanted to throw in another idea re volumes here.

Volumes could also have to do with container placement. For example, if a
volume is specified with multiple containers than the assumption is that
you would want to share data between those containers. Swarm would then
place them on the same host.

A second would be for the volumesfrom flag. This should have similar
functionality.

Otherwise the volume bring requested is going to fail to start for those
volume drivers that cannot share volumes between hosts.

On Sunday, December 13, 2015, Geovani de Souza notifications@github.com
wrote:

+1


Reply to this email directly or view it on GitHub
#1488 (comment).

clintkitson commented Dec 13, 2015

I wanted to throw in another idea re volumes here.

Volumes could also have to do with container placement. For example, if a
volume is specified with multiple containers than the assumption is that
you would want to share data between those containers. Swarm would then
place them on the same host.

A second would be for the volumesfrom flag. This should have similar
functionality.

Otherwise the volume bring requested is going to fail to start for those
volume drivers that cannot share volumes between hosts.

On Sunday, December 13, 2015, Geovani de Souza notifications@github.com
wrote:

+1


Reply to this email directly or view it on GitHub
#1488 (comment).

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Dec 14, 2015

Contributor

@cpuguy83 I think --restart and re-scheduling are incompatible.

For instance, let's say that you start a mysql with --restart=always.

Unless you are using a distributed volume, you definitely DO NOT want Swarm to create a brand new mysql somewhere else with no data. And you definitely do not want for Swarm to destroy that container when the machine finally comes back.

You might want to always restart but never re-schedule, or you might want to get both.

Contributor

aluzzardi commented Dec 14, 2015

@cpuguy83 I think --restart and re-scheduling are incompatible.

For instance, let's say that you start a mysql with --restart=always.

Unless you are using a distributed volume, you definitely DO NOT want Swarm to create a brand new mysql somewhere else with no data. And you definitely do not want for Swarm to destroy that container when the machine finally comes back.

You might want to always restart but never re-schedule, or you might want to get both.

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Dec 14, 2015

Contributor

As an alternative, we would be free to mess around with the restart policies such as --restart=reschedule:on-node-failure.

Contributor

aluzzardi commented Dec 14, 2015

As an alternative, we would be free to mess around with the restart policies such as --restart=reschedule:on-node-failure.

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 14, 2015

Contributor

@aluzzardi Is there something we can add to hint to swarm the scope of the volume (local vs global?)

Contributor

cpuguy83 commented Dec 14, 2015

@aluzzardi Is there something we can add to hint to swarm the scope of the volume (local vs global?)

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Dec 14, 2015

Contributor

@cpuguy83 Well it's totally fine to re-schedule containers that have a local volume if the user says so

Contributor

aluzzardi commented Dec 14, 2015

@cpuguy83 Well it's totally fine to re-schedule containers that have a local volume if the user says so

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 15, 2015

Contributor

In such a case it may be better to only support the explicit case of not rescheduling containers that do have a restart policy.
So, --restart=always -e reschedule=no
This way we can use the restart policy for normal use-cases.

Alternatively, maybe restart policies could be modified to accept conditions like --restart=always,!node-failure.
Engine would continue to care about only the values it currently does (always, on-failure, unless-stopped), and other things in the stack can add their own w/o affecting the actual runtime.

Contributor

cpuguy83 commented Dec 15, 2015

In such a case it may be better to only support the explicit case of not rescheduling containers that do have a restart policy.
So, --restart=always -e reschedule=no
This way we can use the restart policy for normal use-cases.

Alternatively, maybe restart policies could be modified to accept conditions like --restart=always,!node-failure.
Engine would continue to care about only the values it currently does (always, on-failure, unless-stopped), and other things in the stack can add their own w/o affecting the actual runtime.

@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Dec 15, 2015

Contributor

@cpuguy83 The thing is, if the user makes a mistake in setting those flags (or simply doesn't know) we're talking about nuking a production database

Contributor

aluzzardi commented Dec 15, 2015

@cpuguy83 The thing is, if the user makes a mistake in setting those flags (or simply doesn't know) we're talking about nuking a production database

@aluzzardi aluzzardi added this to the 1.1.0 milestone Dec 15, 2015

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 18, 2015

Contributor

@aluzzardi In that case, I'd almost prefer to not reschedule containers with volumes (unless explicitly specified through some configuration) until we can figure out a way to make it just work with restart policies... but maybe there is no perfect world here.

Also wondering if there's a plan to have some delay once a host is marked as unhealthy to do rescheduling (or maybe that's just the health check itself).

Contributor

cpuguy83 commented Dec 18, 2015

@aluzzardi In that case, I'd almost prefer to not reschedule containers with volumes (unless explicitly specified through some configuration) until we can figure out a way to make it just work with restart policies... but maybe there is no perfect world here.

Also wondering if there's a plan to have some delay once a host is marked as unhealthy to do rescheduling (or maybe that's just the health check itself).

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Dec 19, 2015

Contributor

@cpuguy83 by default containers aren't rescheduled at all, volumes or not, it's already opt-in feature.

we could add a delay, what would be the usecase ?

Contributor

vieux commented Dec 19, 2015

@cpuguy83 by default containers aren't rescheduled at all, volumes or not, it's already opt-in feature.

we could add a delay, what would be the usecase ?

@cpuguy83

This comment has been minimized.

Show comment
Hide comment
@cpuguy83

cpuguy83 Dec 19, 2015

Contributor

@vieux I was thinking in favor of using --restart... but you are right.

For the delay, the intention would be to allow the supposedly down node a recovery period.

Contributor

cpuguy83 commented Dec 19, 2015

@vieux I was thinking in favor of using --restart... but you are right.

For the delay, the intention would be to allow the supposedly down node a recovery period.

@HackToday

This comment has been minimized.

Show comment
Hide comment
@HackToday

HackToday Dec 23, 2015

Contributor

@aluzzardi from your suggestion:

Given all the potential issues that might arise by handling the restart policy on the Swarm >side and the fact that duplicate containers may end up running at the same time anyway, >I suggest we do not interfere with --restart and document the fact that rescheduled >containers may be running in parallel for a short time window.

Do you mean the reschedulted containers are allowed to duplicate sometime, and finally, swarm would delete the oldest container, and keep the newest container, is it ?

Contributor

HackToday commented Dec 23, 2015

@aluzzardi from your suggestion:

Given all the potential issues that might arise by handling the restart policy on the Swarm >side and the fact that duplicate containers may end up running at the same time anyway, >I suggest we do not interfere with --restart and document the fact that rescheduled >containers may be running in parallel for a short time window.

Do you mean the reschedulted containers are allowed to duplicate sometime, and finally, swarm would delete the oldest container, and keep the newest container, is it ?

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Dec 23, 2015

Contributor

@HackToday yes, or we might add a flag to decide which one we should keep (newest or oldest)

Contributor

vieux commented Dec 23, 2015

@HackToday yes, or we might add a flag to decide which one we should keep (newest or oldest)

@cblomart

This comment has been minimized.

Show comment
Hide comment
@cblomart

cblomart Dec 30, 2015

speeking of restarts, reschedule and eventually rebalancing seems like speeking of different functionnalities.

the way i look at it after reading these few posts is that rebalancing is another world with considerations like what to do to minimize downtime and volumes access (allow duplicates or not).

restarts and reschedules are the key features that looks like high availability.

restarts might very well be more suitable for statefull services which would require specific volumes.
And it looks to me like reschedules might be more suitable for stateless services.

In the end only the one running the full stack can say what is best.

restrarts are handled at the docker level and reschedules more likely at the swarm level. although if you have different tanks linked together if you add a bucket of water in on, it will naturally spill to the other ones. in this sense docker could check with swarm if it realy is up to him to start a container and eventually leave the job to swarm to decide (certainly not mandatory)

lastly, what i don't grasp is the network implications...
"make sure the new container takes over the ip of the old one"
This certainly looks like a blocking point to start duplicates... idoioticaly the incapability to get back the same ip twice might block the restart of an already rescheduled container.

cblomart commented Dec 30, 2015

speeking of restarts, reschedule and eventually rebalancing seems like speeking of different functionnalities.

the way i look at it after reading these few posts is that rebalancing is another world with considerations like what to do to minimize downtime and volumes access (allow duplicates or not).

restarts and reschedules are the key features that looks like high availability.

restarts might very well be more suitable for statefull services which would require specific volumes.
And it looks to me like reschedules might be more suitable for stateless services.

In the end only the one running the full stack can say what is best.

restrarts are handled at the docker level and reschedules more likely at the swarm level. although if you have different tanks linked together if you add a bucket of water in on, it will naturally spill to the other ones. in this sense docker could check with swarm if it realy is up to him to start a container and eventually leave the job to swarm to decide (certainly not mandatory)

lastly, what i don't grasp is the network implications...
"make sure the new container takes over the ip of the old one"
This certainly looks like a blocking point to start duplicates... idoioticaly the incapability to get back the same ip twice might block the restart of an already rescheduled container.

@dongluochen

This comment has been minimized.

Show comment
Hide comment
@dongluochen

dongluochen Dec 30, 2015

Contributor

I think "make sure the new container takes over the ip of the old one" is unnecessary and may be harmful. Swarm do not specify the IP for the original container. It only attaches the container to an overlay network where IP is dynamically assigned. How this IP is used is up to user. The same logic applies to the new container. Generally speaking, distributed service should use names, not IPs.

Persisting IP usually happens on VM live migration where traffic shouldn't be interrupted. It needs accurate coordination where you have control on both old and new VMs. It's not the case for failure rescheduling.

Contributor

dongluochen commented Dec 30, 2015

I think "make sure the new container takes over the ip of the old one" is unnecessary and may be harmful. Swarm do not specify the IP for the original container. It only attaches the container to an overlay network where IP is dynamically assigned. How this IP is used is up to user. The same logic applies to the new container. Generally speaking, distributed service should use names, not IPs.

Persisting IP usually happens on VM live migration where traffic shouldn't be interrupted. It needs accurate coordination where you have control on both old and new VMs. It's not the case for failure rescheduling.

@ezrasilvera

This comment has been minimized.

Show comment
Hide comment
@ezrasilvera

ezrasilvera Jan 5, 2016

Contributor

Few comments:
We should remember that when the master decides that a node "failed" it doesn't necessarily mean that the node is fully disconnected from the rest of the world!! There might be some partial network failures which might lead (after the rebalancing) to two identical containers running together, this may result in various conflicts and errors. For example, the "failed" node/containers may access the storage storage, causing data corruption, or create an IP address or DNS conflicts. This may also happen due to transient network failures.

In order to safely perform such "rebalancing" the failed node (and/or containers on that node) need first to be "fenced". There are several approaches we can perform such a fence:

  1. Power fence - i.e., shutdown the node
  2. Resource fencing - isolate the node/containers from the storage and network. This is done externally to the node (e.g., fence a node at the Switch it is connected to)
  3. Self fencing - the node itself detects and isolate/shutdown (this approach has many risks and probably doesn't fit the current Docker/Swarm environment)
Contributor

ezrasilvera commented Jan 5, 2016

Few comments:
We should remember that when the master decides that a node "failed" it doesn't necessarily mean that the node is fully disconnected from the rest of the world!! There might be some partial network failures which might lead (after the rebalancing) to two identical containers running together, this may result in various conflicts and errors. For example, the "failed" node/containers may access the storage storage, causing data corruption, or create an IP address or DNS conflicts. This may also happen due to transient network failures.

In order to safely perform such "rebalancing" the failed node (and/or containers on that node) need first to be "fenced". There are several approaches we can perform such a fence:

  1. Power fence - i.e., shutdown the node
  2. Resource fencing - isolate the node/containers from the storage and network. This is done externally to the node (e.g., fence a node at the Switch it is connected to)
  3. Self fencing - the node itself detects and isolate/shutdown (this approach has many risks and probably doesn't fit the current Docker/Swarm environment)
@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Jan 5, 2016

Contributor

@dongluochen alright, let's make this optional

@ezrasilvera you are right, regarding the networking, it should be handled since the container will be disconnected by swarm from the network (we can do this even if the node is unreachable)

Contributor

vieux commented Jan 5, 2016

@dongluochen alright, let's make this optional

@ezrasilvera you are right, regarding the networking, it should be handled since the container will be disconnected by swarm from the network (we can do this even if the node is unreachable)

@ezrasilvera

This comment has been minimized.

Show comment
Hide comment
@ezrasilvera

ezrasilvera Jan 5, 2016

Contributor

One more comment - would we be able to explicitly activate the reschedule functionality for "planned evacuation" (e.g., not as a result of failure) ? This might be helpful for planed maintenance for example.

Contributor

ezrasilvera commented Jan 5, 2016

One more comment - would we be able to explicitly activate the reschedule functionality for "planned evacuation" (e.g., not as a result of failure) ? This might be helpful for planed maintenance for example.

@vieux

This comment has been minimized.

Show comment
Hide comment
@vieux

vieux Jan 5, 2016

Contributor

@ezrasilvera it's planned in the UX but not in the initial PR.

At first we will only support on-node-failure as policy.

We we could definitely imagine on-maintenance if you combine this with the new node management @dongluochen wrote in #1569 we could extend it.

Contributor

vieux commented Jan 5, 2016

@ezrasilvera it's planned in the UX but not in the initial PR.

At first we will only support on-node-failure as policy.

We we could definitely imagine on-maintenance if you combine this with the new node management @dongluochen wrote in #1569 we could extend it.

@vieux vieux modified the milestones: 1.1.0, 1.2.0 Feb 10, 2016

@snoopy86

This comment has been minimized.

Show comment
Hide comment
@snoopy86

snoopy86 Feb 19, 2016

What if you have one node on one location and one on the other and then network fails between them? Would node on the other location also run the container as it would not see the other running container?

snoopy86 commented Feb 19, 2016

What if you have one node on one location and one on the other and then network fails between them? Would node on the other location also run the container as it would not see the other running container?

@michaelzangerle

This comment has been minimized.

Show comment
Hide comment
@michaelzangerle

michaelzangerle Mar 23, 2016

When rescheduling containers, Swarm must handle multi-host networking properly.

Is there anything planned in near future regarding rescheduling and multi-host network?

michaelzangerle commented Mar 23, 2016

When rescheduling containers, Swarm must handle multi-host networking properly.

Is there anything planned in near future regarding rescheduling and multi-host network?

@beverts312

This comment has been minimized.

Show comment
Hide comment
@beverts312

beverts312 Jun 6, 2016

Are there plans to allow an explicit "rebalance" of all eligible containers in the cluster?

  • A container could be considered eligible for a rebalance it they had the reschedule policy on-node-failure or on-explicit (or something like that).
  • This could be invoked via the swarm API

A potential use case would be that we add node(s) to the cluster and want to utilize the newly available resources without having to explicitly choose which containers to go there.

beverts312 commented Jun 6, 2016

Are there plans to allow an explicit "rebalance" of all eligible containers in the cluster?

  • A container could be considered eligible for a rebalance it they had the reschedule policy on-node-failure or on-explicit (or something like that).
  • This could be invoked via the swarm API

A potential use case would be that we add node(s) to the cluster and want to utilize the newly available resources without having to explicitly choose which containers to go there.

@DerekTBrown DerekTBrown referenced this issue Dec 31, 2016

Closed

[FEATURE] Docker Host Scaling #19

2 of 2 tasks complete
@viveky4d4v

This comment has been minimized.

Show comment
Hide comment
@viveky4d4v

viveky4d4v Jan 31, 2017

I am eagerly waiting for this feature , it is really helpful in a clustered environment either small or big .
When can we expect this in docker swarm ?

viveky4d4v commented Jan 31, 2017

I am eagerly waiting for this feature , it is really helpful in a clustered environment either small or big .
When can we expect this in docker swarm ?

@nishanttotla

This comment has been minimized.

Show comment
Hide comment
@nishanttotla

nishanttotla Jan 31, 2017

Contributor

@viveky4d4v are you referring to rebalancing?

Contributor

nishanttotla commented Jan 31, 2017

@viveky4d4v are you referring to rebalancing?

@viveky4d4v

This comment has been minimized.

Show comment
Hide comment
@viveky4d4v

viveky4d4v Feb 1, 2017

@nishanttotla Yes , rebalancing automatically when the node comes back in life . Kind of Resurrection .

Let's say if i have one manager & one worker .-

  • If worker dies , manager will schedule all containers on itself .
  • Worker comes back in life & manager doesn't rebalance containers & all containers sit on manager .
  • Now if manager dies , our stack goes down even if worker is up & running .

Ideally manager should automatically balance the containers when worker comes back in life by moving the oldest container back to the worker . I don't know how this can be done without downtime ( in case you have just one application container ) .
OR
if the scheduler makes some decision like in case of a single application container it should not move it back to worker to remove downtime but it should do it if we have multiple replicas.

PS - I understand it's not efficient way to use swarm, we should use at least 3 managers but i caught up in this situation so i thought to get some ideas from community.

viveky4d4v commented Feb 1, 2017

@nishanttotla Yes , rebalancing automatically when the node comes back in life . Kind of Resurrection .

Let's say if i have one manager & one worker .-

  • If worker dies , manager will schedule all containers on itself .
  • Worker comes back in life & manager doesn't rebalance containers & all containers sit on manager .
  • Now if manager dies , our stack goes down even if worker is up & running .

Ideally manager should automatically balance the containers when worker comes back in life by moving the oldest container back to the worker . I don't know how this can be done without downtime ( in case you have just one application container ) .
OR
if the scheduler makes some decision like in case of a single application container it should not move it back to worker to remove downtime but it should do it if we have multiple replicas.

PS - I understand it's not efficient way to use swarm, we should use at least 3 managers but i caught up in this situation so i thought to get some ideas from community.

@nishanttotla

This comment has been minimized.

Show comment
Hide comment
@nishanttotla

nishanttotla Feb 1, 2017

Contributor

@viveky4d4v I want to confirm that you mean Docker Swarm standalone (this project docker/swarm) and not the new Swarm mode released in Docker 1.12. The manager/worker terminology, and running container replicas (services) is a feature of Swarm mode, not this project.

Contributor

nishanttotla commented Feb 1, 2017

@viveky4d4v I want to confirm that you mean Docker Swarm standalone (this project docker/swarm) and not the new Swarm mode released in Docker 1.12. The manager/worker terminology, and running container replicas (services) is a feature of Swarm mode, not this project.

@viveky4d4v

This comment has been minimized.

Show comment
Hide comment
@viveky4d4v

viveky4d4v Feb 2, 2017

@nishanttotla : Yes , you are correct . MY BAD !

I will raise the issue at "https://github.com/docker/docker/issues/new"

viveky4d4v commented Feb 2, 2017

@nishanttotla : Yes , you are correct . MY BAD !

I will raise the issue at "https://github.com/docker/docker/issues/new"

@piotrminkina

This comment has been minimized.

Show comment
Hide comment
@piotrminkina

piotrminkina commented Oct 30, 2017

I think this issue is implemented. See https://docs.docker.com/swarm/scheduler/rescheduling/

@nishanttotla

This comment has been minimized.

Show comment
Hide comment
@nishanttotla

nishanttotla Oct 30, 2017

Contributor

@piotrminkina right, I think we can close this issue. New issues can and should be opened for issues with rescheduling.

Contributor

nishanttotla commented Oct 30, 2017

@piotrminkina right, I think we can close this issue. New issues can and should be opened for issues with rescheduling.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment