New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improve CircleCI branch failure messaging #1552

Merged
merged 10 commits into from Jan 4, 2019

Conversation

2 participants
@chrisgilmerproj
Copy link
Contributor

chrisgilmerproj commented Jan 4, 2019

Description

The current CircleCI messaging to slack and pager duty is inadequate in that there isn't enough context. This update should improve the context we see around build failures in CircleCI.

New message will look like this:

screen shot 2019-01-03 at 5 05 27 pm

Also, if a job has the word deploy in it then the color will change to red.

Reviewer Notes

Please review the messaging to ensure it meets our expectations.

Setup

The way I tested this was to comment out the code that sends notifications to PagerDuty. Then I modified the .circleci/config.yml so that build_tools would fail early and announce_failures would trigger only on my branch. I let people in #dp3-engineering know that this was happening in advance and then pushed my work. That helped me create the image above.

All of this work will have to be reverted before merging.

Code Review Verification Steps

  • Request review from a member of a different team.
  • Have the Pivotal acceptance criteria been met for this change?

References

@chrisgilmerproj chrisgilmerproj self-assigned this Jan 4, 2019

@@ -39,7 +39,7 @@ commands:
- run:
name: Announce failure
command: |
[[ $CIRCLE_BRANCH = master ]] || exit 0
[[ $CIRCLE_BRANCH = cg_162974053_improve_slack_failure_message ]] || exit 0

This comment has been minimized.

@chrisgilmerproj

chrisgilmerproj Jan 4, 2019

Contributor

Must revert!

@@ -406,6 +406,7 @@ jobs:
executor: mymove_small
steps:
- checkout
- run: exit 1

This comment has been minimized.

@chrisgilmerproj

chrisgilmerproj Jan 4, 2019

Contributor

Must revert!

# -H "Accept: application/vnd.pagerduty+json;version=2" \
# -H "Content-Type: application/json" \
# --data "$pd_payload" \
# "https://events.pagerduty.com/v2/enqueue"

This comment has been minimized.

@chrisgilmerproj

chrisgilmerproj Jan 4, 2019

Contributor

Must uncomment!

@@ -1,31 +1,47 @@
#!/bin/bash
set -euo pipefail

GIT_TEXT=$(git --no-pager show --format=format:'%s')

This comment has been minimized.

@chrisgilmerproj

chrisgilmerproj Jan 4, 2019

Contributor

This turns out to have no value AND because we don't escape the double quotes it can invalidate the data package we send to Slack, which means we'd not see any alerts that included diffs with double quotes.

"severity": "info",
"class": "cicd failure"
},
"routing_key": "$PD_ROUTING_KEY",
"dedup_key": "circle-$CIRCLE_BUILD_NUM",
"dedup_key": "circle-$CIRCLE_WORKFLOW_ID",

This comment has been minimized.

@chrisgilmerproj

chrisgilmerproj Jan 4, 2019

Contributor

The workflow ID is the same across all jobs, which means if multiple jobs fail for a branch then the on-call will only get one page instead of multiple.

This comment has been minimized.

@kahlouie

kahlouie Jan 4, 2019

Contributor

yay!!!!

@kahlouie
Copy link
Contributor

kahlouie left a comment

This looks great! When you revert the revertables, I will approve!

"fallback": "$message $CIRCLE_BUILD_URL",
"color": "$color",
"pretext": "$pretext",
"author_name": "$CIRCLE_USERNAME",

This comment has been minimized.

@kahlouie

kahlouie Jan 4, 2019

Contributor

I like the addition of the circle username to figure out who has the most context!

"severity": "info",
"class": "cicd failure"
},
"routing_key": "$PD_ROUTING_KEY",
"dedup_key": "circle-$CIRCLE_BUILD_NUM",
"dedup_key": "circle-$CIRCLE_WORKFLOW_ID",

This comment has been minimized.

@kahlouie

kahlouie Jan 4, 2019

Contributor

yay!!!!

@chrisgilmerproj

This comment has been minimized.

Copy link
Contributor

chrisgilmerproj commented Jan 4, 2019

@kahlouie - I did the revert! :)

@chrisgilmerproj chrisgilmerproj merged commit d6baddf into master Jan 4, 2019

12 checks passed

ci/circleci: acceptance_tests Your tests passed on CircleCI!
Details
ci/circleci: build_app Your tests passed on CircleCI!
Details
ci/circleci: build_migrations Your tests passed on CircleCI!
Details
ci/circleci: build_tools Your tests passed on CircleCI!
Details
ci/circleci: client_test Your tests passed on CircleCI!
Details
ci/circleci: integration_tests_mymove Your tests passed on CircleCI!
Details
ci/circleci: integration_tests_office Your tests passed on CircleCI!
Details
ci/circleci: integration_tests_tsp Your tests passed on CircleCI!
Details
ci/circleci: pre_deps_golang Your tests passed on CircleCI!
Details
ci/circleci: pre_deps_yarn Your tests passed on CircleCI!
Details
ci/circleci: pre_test Your tests passed on CircleCI!
Details
ci/circleci: server_test Your tests passed on CircleCI!
Details

@chrisgilmerproj chrisgilmerproj deleted the cg_162974053_improve_slack_failure_message branch Jan 4, 2019

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment