Downtime Post-Mortem - 8/24/2012 - 9:40AM PDT - 11:15AM PDT

[mahmoudimus]

At approximately 9:40AM PDT, Balanced's centralized logging machines experienced slowness as their free disk space started approaching dangerous levels. We are currently investigating why our alerting systems did not notify us at a lower threshold. Balanced's servers were operating normally at starting 11:15AM PDT.

Internally, we centralize our logs through the industry standard rsyslog, which is supposed to spool logs locally and asynchronously deliver them to our centralized logging server. However, a misconfiguration in our rsyslog configuration caused log messages to synchronously deliver instead of asynchronously, consequently impacting the responsiveness of our API applications.

Yesterday, we added two new machines to help with the current load we've been experiencing on Balanced, but we did not anticipate the surge in logs generated.

We're going to write sanity checks for our rsyslog configurations and install a redundant alerting system to allow us to respond faster to these issues before they arise. We're also now correlating the increase of log messages to a particular server, to monitor exactly how much disk space usage accrues per new API machine.

We understand that this is unacceptable and we're working tirelessly to ensure Balanced is operating efficiently and smoothly.

[mahmoudimus]

#12 and #39 are relevant here.