Postmortem & rebuild server

[bjacobel]

[DigitalOcean] New Ticket # 767564 : Networking disabled: citadel

Hi there,

We are sorry to report that we have detected what appears to be a large flood of traffic from one or more of your servers that is disrupting the normal traffic flow for other users.

I got owned again :( early signs point to unsecured Elasticsearch -- right before New Relic cut out there was a huge CPU spike caused by some process I don't recognize running under the elasticsearch user.

• Find out how they got in
• find out how to stop it from happening again
• Get rid of the droplet
• Spin up new server (Could be an opportunity to switch to AWS)
• reconnect DNS
• rebuild server (bright side: good full test of Ansible)
• bonus: Get elasticsearch off the server and switch to Amazon's new hosted ES