- 12factor.net : A great write up by Heroku on making an app scalable, modular, and easy to upgrade. Basically, how to build apps in the new world.
- https://github.com/phrawzty/hiera-s3: Nice hiera plugin from phrawzty (but maybe just use vault! :) )
- Visible Ops Book: http://www.amazon.com/The-Visible-Ops-Handbook-Implementing/dp/0975568612
- Phoenix Project Book: http://www.amazon.com/The-Phoenix-Project-Helping-Business/dp/0988262592
- Our Terraform code: https://github.com/mozilla/socorro-infra/blob/master/terraform
- Our Puppet code: https://github.com/mozilla/socorro-infra/blob/master/puppet
- Our rustic deployment script: https://github.com/mozilla/socorro-infra/blob/master/bin/deploy-socorro.sh
- Our Packer code: https://github.com/mozilla/socorro-infra/blob/master/packer
- Our user data code: https://github.com/mozilla/socorro-infra/blob/master/terraform/webapp/socorro_role.sh
- A cool way to cycle through dashboards: http://dashplay.io/0p98yofx+
- Our current events: https://p.datadoghq.com/sb/282da3b8c-2916c1621c
- Today vs yesterday: https://p.datadoghq.com/sb/282da3b8c-3c0ed4ed8d
- Prod stats: https://p.datadoghq.com/sb/282da3b8c-bfee0403b5
- How to send custom and arbitrary metrics to cloudwatch (you can track # of coffees consumed, or anything else) https://www.loggly.com/blog/send-custom-metrics-to-cloudwatchs-api/
- Packer can create an AMI for any region, and even create an AMI for multiple regions. Super useful for deployment!
- US Standard is simply (US-East-1 and US-West-1) in failover mode. This is why things in US Standard are 'eventual consistency' for reads after writes.
- If you need immediate, reliable consistency, choose a specific region when creating the bucket.
- US-Standard does not cost extra compared to single-region non-redundant S3 regions.
- Cloudfront costs the same per request as S3.
- You do get SSL with S3, but not on http endpoints. ( Here, have this page explain it to you...it's a little weird. http://docs.aws.amazon.com/general/latest/gr/rande.html )
- Otherwise, use SSL in cloudfront. Use the old IAM tools to upload it...
- Our naming standards for AWS infrastructure(do this ASAP!): https://dl.dropboxusercontent.com/u/2273146/aws-naming-conventions.txt
- When given a choice, download/request your SSL certs in Apache format. This goes right up to AWS easily.
iam-servercertupload -b ./mysite.crt -k ./mysite.key -c ./intermediatechain.crt -p /cloudfront/ -s mysite-2015-09-25
- Always use autoscaling groups for everything, even things that never need to scale. You always want 1 of some server up...autoscaling groups make it so if that one dies, a new one automatically replaces it.
- You can have multiple autoscaling groups scale into a single ELB.
- You can have a single autoscaling group add/remove servers from multiple ELBs.
- These concepts are useful if you want to balance out the # of instance store vs. EBS backed EC2 instances...EBS is usually the culprit on AWS @#$@storm days
- Go multi-region early rather than later...it doesn't get easier the longer you wait!
- When load testing or expecting heavy traffic, it's a great idea to call AWS support and have them "prewarm" your ELB. ELBs are basically just EC2 instances running load balancing software, and 'prewarming' them is scaling them up.
- To get great ratings for SSL security, check out Mozilla's SSL cipher tool (credit to Gene!) https://mozilla.github.io/server-side-tls/ssl-config-generator/
- If you have to have multiple AZ's in staging but only have one node up, enable cross zone balancing.
- CNAME everything! Let's say you have a database server: webappdatabase.rdsaddress.2342342342.aws.com... If you cname that to webapp-db.internaldomain.com instead, if you ever have to switch databases or regions you have a super easy way to switch over!
- Health check based routing is awesome, as is geo-ip routing.
- Decide on a tagging strategy early on....Mozilla Tools and Services team uses: environment, role, project. These tags play a major role in monitoring, metrics, and even permissions.
- Always, immediately, first thing...enable Cloudtrail in AWS.
- Setup an IFTTT recipe which will monitor the status RSS feeds for AWS, and SMS/email/alert you and your team when those RSS feeds get updated.
- Be sure you have multiple sources watching your apps!
- Easy 'pound one address' tool: bees with machineguns (it's awesome to make an endpoint that exercises the entire app/funnel/workflow that you can then pound on with bees...and the logs are so fun to read)
- Setup detailed billing and a bucket to store it im
- Try out Netflix ICE cost analysis software
- When you know you need an instance for at least a year, setup reserved instances
- A Great Intro to 'Gamedays' (The Way to Make Apps and Teams More Resilient by Dylan Richard): https://www.youtube.com/watch?v=LCZT_Q3z520
ABCD: ALWAYS BE CHECKING DATADOG