New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IsTaskDirty: Ignore PullOptions for running tasks #2351

Merged
merged 1 commit into from Aug 17, 2017

Conversation

Projects
None yet
6 participants
@jlhawn
Contributor

jlhawn commented Aug 14, 2017

This patch causes the orchestrator to ignore the PullOptions field of a ContainerSpec when determining whether a task is considered to be 'dirty'. It is only ignored if and only if the current state of the task is either READY, STARTING, or RUNNING.

Related to #971

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Aug 14, 2017

Contributor

Only do this at the point in the orchestrator where tasks are being compared. Doing this globally will break other update functionality, such as the dispatcher pushes and agent task manager updates (probably okay if that one is ignored).

Contributor

stevvooe commented Aug 14, 2017

Only do this at the point in the orchestrator where tasks are being compared. Doing this globally will break other update functionality, such as the dispatcher pushes and agent task manager updates (probably okay if that one is ignored).

@jlhawn

This comment has been minimized.

Show comment
Hide comment
@jlhawn

jlhawn Aug 14, 2017

Contributor

@stevvooe I'm not familiar enough with this repo to know where the point in the orchestrator where tasks are being compared is. A quick search of the codebase for calls to equality.TasksEqualStable yields these 3 locations:

if equality.TasksEqualStable(task, tm.task) {

if equality.TasksEqualStable(oldTask, t) && t.Status.State > api.TaskStateAssigned {

if equality.TasksEqualStable(oldTask, v.Task) && v.Task.Status.State > api.TaskStateAssigned {

Are any of these 3 the location you were thinking of?

Contributor

jlhawn commented Aug 14, 2017

@stevvooe I'm not familiar enough with this repo to know where the point in the orchestrator where tasks are being compared is. A quick search of the codebase for calls to equality.TasksEqualStable yields these 3 locations:

if equality.TasksEqualStable(task, tm.task) {

if equality.TasksEqualStable(oldTask, t) && t.Status.State > api.TaskStateAssigned {

if equality.TasksEqualStable(oldTask, v.Task) && v.Task.Status.State > api.TaskStateAssigned {

Are any of these 3 the location you were thinking of?

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Aug 14, 2017

Contributor

There are two ways:

  1. https://github.com/docker/swarmkit/blob/master/manager/controlapi/service.go#L728 would allow checking whether the update is actually a change and ignore it, right at the AP.

  2. https://github.com/docker/swarmkit/blob/master/manager/orchestrator/task.go#L62 would influence whether the orchestrator believes the task to be "dirty" and issue updates if that is the case. Check usages there to see if it makes sense.

Number 1 is kind of a hack. Number 2 seems to have some caveats but seems doable.

Let me know if that is enough to go on.

Contributor

stevvooe commented Aug 14, 2017

There are two ways:

  1. https://github.com/docker/swarmkit/blob/master/manager/controlapi/service.go#L728 would allow checking whether the update is actually a change and ignore it, right at the AP.

  2. https://github.com/docker/swarmkit/blob/master/manager/orchestrator/task.go#L62 would influence whether the orchestrator believes the task to be "dirty" and issue updates if that is the case. Check usages there to see if it makes sense.

Number 1 is kind of a hack. Number 2 seems to have some caveats but seems doable.

Let me know if that is enough to go on.

@codecov

This comment has been minimized.

Show comment
Hide comment
@codecov

codecov bot Aug 14, 2017

Codecov Report

Merging #2351 into master will increase coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2351      +/-   ##
==========================================
+ Coverage   59.96%   60.04%   +0.07%     
==========================================
  Files         128      128              
  Lines       26183    26183              
==========================================
+ Hits        15701    15721      +20     
+ Misses       9092     9066      -26     
- Partials     1390     1396       +6

codecov bot commented Aug 14, 2017

Codecov Report

Merging #2351 into master will increase coverage by 0.07%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #2351      +/-   ##
==========================================
+ Coverage   59.96%   60.04%   +0.07%     
==========================================
  Files         128      128              
  Lines       26183    26183              
==========================================
+ Hits        15701    15721      +20     
+ Misses       9092     9066      -26     
- Partials     1390     1396       +6

@jlhawn jlhawn changed the title from Task Equality: Ignore container image pull options to Orchestrator: IsTaskDirty: Ignore PullOptions Aug 14, 2017

@jlhawn

This comment has been minimized.

Show comment
Hide comment
@jlhawn

jlhawn Aug 14, 2017

Contributor

@stevvooe I've updated this patch to only apply to the orchestrator.IsTaskDirty function (which is used by the updater).

Contributor

jlhawn commented Aug 14, 2017

@stevvooe I've updated this patch to only apply to the orchestrator.IsTaskDirty function (which is used by the updater).

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Aug 14, 2017

Contributor

LGTM

Make sure to test this one thoroughly.

Contributor

stevvooe commented Aug 14, 2017

LGTM

Make sure to test this one thoroughly.

@aaronlehmann

This comment has been minimized.

Show comment
Hide comment
@aaronlehmann

aaronlehmann Aug 15, 2017

Collaborator

I don't think this is a good idea. With this change, if you deployed a service with incorrect credentials, updating those credentials would not necessarily correct the problem.

It would happen to work with a restart policy of "always" or "on failure", because the service update would trigger a reconciliation, and the orchestrator would start new tasks with the updated spec. But if the restart policy is "never" or the service is running up against the restart limit, the orchestrator will check if anything has changed in the service spec to merit forcing a restart, and this hack will tell it that nothing has changed.

There are workarounds for this, such as making this hack apply only to tasks with DesiredState <= api.TaskStateRunning (that way, a change to credentials won't replace a running task, but will cause a dead task to be replaced). But I find this hack on top of a hack quite unpleasant. My preference would be to keep things simple and say that if you change anything in ContainerSpec, that will always kick off a rolling update. Ideally, we should avoid rotating credentials frequently, and/or set up rolling update parameters so that the rolling updates aren't disruptive.

But if this must be done, I suppose the extra hack I mentioned would work.

If you do move forward with this, I'd recommend writing a test that runs with various restart policies, starting with a 1-replica service and an up-to-date task with desired state "shutdown", updating the service to change PullOptions, and confirming that this change causes a new task to be created.

Collaborator

aaronlehmann commented Aug 15, 2017

I don't think this is a good idea. With this change, if you deployed a service with incorrect credentials, updating those credentials would not necessarily correct the problem.

It would happen to work with a restart policy of "always" or "on failure", because the service update would trigger a reconciliation, and the orchestrator would start new tasks with the updated spec. But if the restart policy is "never" or the service is running up against the restart limit, the orchestrator will check if anything has changed in the service spec to merit forcing a restart, and this hack will tell it that nothing has changed.

There are workarounds for this, such as making this hack apply only to tasks with DesiredState <= api.TaskStateRunning (that way, a change to credentials won't replace a running task, but will cause a dead task to be replaced). But I find this hack on top of a hack quite unpleasant. My preference would be to keep things simple and say that if you change anything in ContainerSpec, that will always kick off a rolling update. Ideally, we should avoid rotating credentials frequently, and/or set up rolling update parameters so that the rolling updates aren't disruptive.

But if this must be done, I suppose the extra hack I mentioned would work.

If you do move forward with this, I'd recommend writing a test that runs with various restart policies, starting with a 1-replica service and an up-to-date task with desired state "shutdown", updating the service to change PullOptions, and confirming that this change causes a new task to be created.

@andrewhsu

This comment has been minimized.

Show comment
Hide comment
@andrewhsu

andrewhsu Aug 15, 2017

Is it possible to have a test case for this in this repo? If not for the test case described by @aaronlehmann at least something that can verify an expected change in behaviour with this patch that is desirable?

andrewhsu commented Aug 15, 2017

Is it possible to have a test case for this in this repo? If not for the test case described by @aaronlehmann at least something that can verify an expected change in behaviour with this patch that is desirable?

@jlhawn

This comment has been minimized.

Show comment
Hide comment
@jlhawn

jlhawn Aug 15, 2017

Contributor

@andrewhsu I can throw together a unit test for the IsTaskDirty function.

Contributor

jlhawn commented Aug 15, 2017

@andrewhsu I can throw together a unit test for the IsTaskDirty function.

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Aug 15, 2017

Contributor

@aaronlehmann Thanks for providing input here!

This was definitely a concern I brought up. Looking at the various call sites for IsTaskDirty, it looks like there might be a spot where this can have the desired effect without creating the described impact. Is there a specific spot you had in mind?

@andrewhsu Yes, these situations should be testable in this repo.

@andrewhsu I can throw together a unit test for the IsTaskDirty function.

@jlhawn Based on @aaronlehmann's feedback, I am not quite sure that is enough. There are cases where we want the dirty task to propagate and cases where we don't. It seems like we just need to do this comparison only at certain callsites.

Contributor

stevvooe commented Aug 15, 2017

@aaronlehmann Thanks for providing input here!

This was definitely a concern I brought up. Looking at the various call sites for IsTaskDirty, it looks like there might be a spot where this can have the desired effect without creating the described impact. Is there a specific spot you had in mind?

@andrewhsu Yes, these situations should be testable in this repo.

@andrewhsu I can throw together a unit test for the IsTaskDirty function.

@jlhawn Based on @aaronlehmann's feedback, I am not quite sure that is enough. There are cases where we want the dirty task to propagate and cases where we don't. It seems like we just need to do this comparison only at certain callsites.

@jlhawn

This comment has been minimized.

Show comment
Hide comment
@jlhawn

jlhawn Aug 15, 2017

Contributor

There are workarounds for this, such as making this hack apply only to tasks with DesiredState <= api.TaskStateRunning (that way, a change to credentials won't replace a running task, but will cause a dead task to be replaced).

Why would we want to look at the desired state of the task? Wouldn't we only want to ignore the PullOptions field if-and-only-if the current state of the task is api.TaskStateRunning?

Contributor

jlhawn commented Aug 15, 2017

There are workarounds for this, such as making this hack apply only to tasks with DesiredState <= api.TaskStateRunning (that way, a change to credentials won't replace a running task, but will cause a dead task to be replaced).

Why would we want to look at the desired state of the task? Wouldn't we only want to ignore the PullOptions field if-and-only-if the current state of the task is api.TaskStateRunning?

@stevvooe

This comment has been minimized.

Show comment
Hide comment
@stevvooe

stevvooe Aug 15, 2017

Contributor

Why would we want to look at the desired state of the task? Wouldn't we only want to ignore the PullOptions field if-and-only-if the current state of the task is api.TaskStateRunning?

He is not suggesting that you look at the desired state. He is saying that the logic should be outside of IsTaskDirty because we don't want these changes to always be ignored for the dirty test.

Contributor

stevvooe commented Aug 15, 2017

Why would we want to look at the desired state of the task? Wouldn't we only want to ignore the PullOptions field if-and-only-if the current state of the task is api.TaskStateRunning?

He is not suggesting that you look at the desired state. He is saying that the logic should be outside of IsTaskDirty because we don't want these changes to always be ignored for the dirty test.

@aaronlehmann

This comment has been minimized.

Show comment
Hide comment
@aaronlehmann

aaronlehmann Aug 15, 2017

Collaborator
Collaborator

aaronlehmann commented Aug 15, 2017

@jlhawn jlhawn changed the title from Orchestrator: IsTaskDirty: Ignore PullOptions to Updater.isTaskDirty: Ignore PullOptions Aug 16, 2017

@jlhawn jlhawn changed the title from Updater.isTaskDirty: Ignore PullOptions to IsTaskDirty: Ignore PullOptions for running tasks Aug 16, 2017

Show outdated Hide outdated manager/orchestrator/task.go Outdated
Show outdated Hide outdated manager/orchestrator/task.go Outdated
@aluzzardi

This comment has been minimized.

Show comment
Hide comment
@aluzzardi

aluzzardi Aug 16, 2017

Contributor

LGTM, couple of nits

Contributor

aluzzardi commented Aug 16, 2017

LGTM, couple of nits

IsTaskDirty: Ignore PullOptions for running tasks
This patch causes the orchestrator to ignore the PullOptions field of a
ContainerSpec when determining whether a task is considered to be 'dirty'.
It is only ignored if and only if the current state of the task is either
READY, STARTING, or RUNNING.

Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)

@aluzzardi aluzzardi merged commit 163a8c2 into docker:master Aug 17, 2017

3 checks passed

ci/circleci Your tests passed on CircleCI!
Details
codecov/project 60.04% (target 0%)
Details
dco-signed All commits are signed
@@ -67,7 +67,29 @@ func IsTaskDirty(s *api.Service, t *api.Task) bool {
return false
}
return !reflect.DeepEqual(s.Spec.Task, t.Spec) ||
// Make a deep copy of the service and task spec for the comparison.
serviceTaskSpec := *s.Spec.Task.Copy()

This comment has been minimized.

@aluzzardi

aluzzardi Aug 17, 2017

Contributor

Actually, @jlhawn, shouldn't we not dereference the pointer here now that we're doing a deep copy?

@aluzzardi

aluzzardi Aug 17, 2017

Contributor

Actually, @jlhawn, shouldn't we not dereference the pointer here now that we're doing a deep copy?

This comment has been minimized.

@jlhawn

jlhawn Aug 17, 2017

Contributor

We discussed this offline but posting here to follow-up: The call to DeepEqual() later on is comparing it to a non-pointer api.TaskSpec. I could have either dereferenced it here or there, but what's important is that it is still calling DeepEqual() with the same types.

@jlhawn

jlhawn Aug 17, 2017

Contributor

We discussed this offline but posting here to follow-up: The call to DeepEqual() later on is comparing it to a non-pointer api.TaskSpec. I could have either dereferenced it here or there, but what's important is that it is still calling DeepEqual() with the same types.

@jlhawn jlhawn deleted the jlhawn:temp_971 branch Aug 17, 2017

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment