Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better cleanup of incremental tasks #1189

Merged
merged 3 commits into from
Aug 22, 2016
Merged

Conversation

ssalinas
Copy link
Member

@ssalinas ssalinas commented Aug 3, 2016

This PR addresses two bugs:

  • When an incremental deploys fails, it will wait for the active deploy to return to normal instance count so there is never reduced capacity in a load balancer. If there are not enough resources to return the active deploy to the original instance count, these can essentially be 'stuck' in cleaning state waiting for more tasks to start.
  • If bounce is initiated during a deploy that will fail and the replacement tasks for the bounce are continually failing healthchecks, the failed deploy tasks will not shut down and will remain in cleaning until the bounce completes (not harmful, but annoying since they should be killed more quickly)

These are both due to the change introduced in d6ba7e3 which considered deploy failures and cancellations as non-immediate kills for the sake of incremental deploy rollbacks as mentioned in the first point above.

This PR updates the deploy checker and cleaner to be more aware of incremental vs non-incremental deploy failures so that the cleanup can be done appropriately, as well as cleaning up tasks from failed incremental deploys in a more rolling fashion (similar to incremental bounce)

/cc @tpetr

@ssalinas ssalinas changed the title (WIP) Better cleanup of incremental tasks Better cleanup of incremental tasks Aug 3, 2016
@ssalinas ssalinas modified the milestone: 0.10.0 Aug 4, 2016
@ssalinas ssalinas modified the milestones: 0.10.0, 0.11.0, 0.10.1 Aug 19, 2016
@ssalinas
Copy link
Member Author

Merging this for 0.10.1 bug fix release

@ssalinas ssalinas merged commit 591723a into master Aug 22, 2016
@ssalinas ssalinas deleted the incremental_task_cleanups branch August 22, 2016 15:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant