New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Better cleanup of incremental tasks #1189

Merged
merged 3 commits into from Aug 22, 2016

Conversation

Projects
None yet
1 participant
@ssalinas
Member

ssalinas commented Aug 3, 2016

This PR addresses two bugs:

  • When an incremental deploys fails, it will wait for the active deploy to return to normal instance count so there is never reduced capacity in a load balancer. If there are not enough resources to return the active deploy to the original instance count, these can essentially be 'stuck' in cleaning state waiting for more tasks to start.
  • If bounce is initiated during a deploy that will fail and the replacement tasks for the bounce are continually failing healthchecks, the failed deploy tasks will not shut down and will remain in cleaning until the bounce completes (not harmful, but annoying since they should be killed more quickly)

These are both due to the change introduced in d6ba7e3 which considered deploy failures and cancellations as non-immediate kills for the sake of incremental deploy rollbacks as mentioned in the first point above.

This PR updates the deploy checker and cleaner to be more aware of incremental vs non-incremental deploy failures so that the cleanup can be done appropriately, as well as cleaning up tasks from failed incremental deploys in a more rolling fashion (similar to incremental bounce)

/cc @tpetr

ssalinas added some commits Aug 3, 2016

@ssalinas ssalinas changed the title from (WIP) Better cleanup of incremental tasks to Better cleanup of incremental tasks Aug 3, 2016

@ssalinas ssalinas added the hs_staging label Aug 4, 2016

@ssalinas ssalinas modified the milestone: 0.10.0 Aug 4, 2016

@ssalinas ssalinas modified the milestones: 0.10.0, 0.11.0, 0.10.1 Aug 19, 2016

@ssalinas

This comment has been minimized.

Show comment
Hide comment
@ssalinas

ssalinas Aug 22, 2016

Member

Merging this for 0.10.1 bug fix release

Member

ssalinas commented Aug 22, 2016

Merging this for 0.10.1 bug fix release

@ssalinas ssalinas merged commit 591723a into master Aug 22, 2016

1 of 2 checks passed

continuous-integration/travis-ci/pr The Travis CI build could not complete due to an error
Details
continuous-integration/travis-ci/push The Travis CI build passed
Details

@ssalinas ssalinas deleted the incremental_task_cleanups branch Aug 22, 2016

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment