This repository has been archived by the owner on Aug 27, 2019. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 3
2016 03 29 API server degradation postmortem timeline
M. Adam Kendall edited this page Mar 29, 2016
·
4 revisions
ALL TIMES IN EDT
- 1144 CF 233 build fails in Concourse, which leaves
api_z1/0VM in down state: https://18f.slack.com/archives/cloud-gov-ops/p1459266254001029 - 1148 Adam find initial cause to be database migration https://18f.slack.com/archives/cloud-gov-ops/p1459266499001035
- 1152 Vraj Mohan reports issues with pushing apps https://18f.slack.com/archives/cloud-gov-support/p1459266765000562
- 1158 Josh Carp clarifies that issues are 502 errors from API https://18f.slack.com/archives/cloud-gov-support/p1459267092000570
- 1202 Adam explains issue due to failure in API upgrade process https://18f.slack.com/archives/cloud-gov-support/p1459267370000572
- 1204 Schema migration found that is root cause of the issue https://18f.slack.com/archives/cloud-gov-ops/p1459267533001049
- 1210 Resolved through edit of the migration file directly on the API vm that previous failed and Concourse job restarted
- 1230 Github issue filed with CloudFoundry https://github.com/cloudfoundry/cloud_controller_ng/issues/570
- 1559 Bret points out that statuspage.io should have been updated for more visibility into the issue https://18f.slack.com/archives/cloud-gov-support/p1459281560000574
- 1619 Adam updates statuspage with resolved incident details: https://cloudgov.statuspage.io/incidents/r2hw8g5x08nz