Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve Release Failed Webhook Messages #1965

Closed
wants to merge 4 commits into from
Closed

Improve Release Failed Webhook Messages #1965

wants to merge 4 commits into from

Conversation

awsmsrc
Copy link
Contributor

@awsmsrc awsmsrc commented Feb 23, 2017

We are incorrectly sending release failed webhooks with a message that says unable to wait for promotion: ResourceNotReady: exceeded 120 wait attempts.

Looking at the aws Waiter source code it returns an error when the stack is in an update failed state, we were not handling this properly and always outputting this error.

This PR

  • Increases the waiter 3x to cover observations that a stack can take more than an hour to rollback
  • Changes the message include "UPDATE_FAILED", "UPDATE_ROLLBACK_FAILED", "UPDATE_ROLLBACK_COMPLETE" and event status correctly
  • adds more info to the error message: message": "release RYSPOEAUOVY failed - [AWS::ECS::Service:ServiceWeb] [UPDATE_FAILED]: Resource update cancelled",

@awsmsrc awsmsrc changed the title first pass concept at increasing waiter time [WIP] first pass concept at increasing waiter time Feb 23, 2017
@nzoschke
Copy link
Contributor

nzoschke commented Feb 23, 2017

QA Checklist

  • Create an app
  • Deploy an app with a load balancer does an exit 1
  • Confirm Release Promote Started and Release Promote Failed webhook and "UPDATE_ROLLBACK_COMPLETE" message

Copy link
Contributor

@nzoschke nzoschke left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code LGTM

Complete the QA checklist before merge

@codecov
Copy link

codecov bot commented Feb 23, 2017

Codecov Report

Merging #1965 into master will decrease coverage by -0.02%.
The diff coverage is 0%.

@@            Coverage Diff             @@
##           master    #1965      +/-   ##
==========================================
- Coverage   30.16%   30.15%   -0.02%     
==========================================
  Files         140      140              
  Lines       13986    13995       +9     
==========================================
+ Hits         4219     4220       +1     
- Misses       9323     9331       +8     
  Partials      444      444
Impacted Files Coverage Δ
api/models/release.go 0% <ø> (ø)
api/models/cronjob.go 86.2% <ø> (+3.44%)

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update eab9393...0a01dc0. Read the comment docs.

@nzoschke nzoschke changed the title [WIP] first pass concept at increasing waiter time [WIP] Improve Release Failed Webhook Messages Feb 23, 2017
@awsmsrc
Copy link
Contributor Author

awsmsrc commented Feb 23, 2017

@MiguelMoll manual QA complete, please review

Copy link
Contributor

@MiguelMoll MiguelMoll left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm

@awsmsrc awsmsrc changed the title [WIP] Improve Release Failed Webhook Messages Improve Release Failed Webhook Messages Feb 23, 2017
@MiguelMoll MiguelMoll added this to the 20170223 milestone Feb 23, 2017
MiguelMoll added a commit that referenced this pull request Feb 23, 2017
@MiguelMoll MiguelMoll mentioned this pull request Feb 23, 2017
18 tasks
@awsmsrc
Copy link
Contributor Author

awsmsrc commented Feb 27, 2017

fixes #1654

SonOfBytes pushed a commit to SonOfBytes/rack that referenced this pull request Mar 17, 2017
@ddollar ddollar deleted the retry_waiter branch March 29, 2017 22:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
3 participants