Update stack for safecastrpt-prd (https://grafana.safecast.cc/)

[matschaffer]

Got an email from AWS that the stack safecastrpt-prd (https://grafana.safecast.cc/) is running on is no longer getting updates.

Should be able to copy the config and create a new environment on the latest stack.

Note that this stack is currently "64bit Amazon Linux 2018.03 v2.9.1 running PHP 5.6" but doesn't actually use PHP.

Just reached for that as a fairly generic stack with a webserver that provides virtualhost support.

Could probably use a variety of stacks, but watch out not to use nginx, or if you do make sure to update stuff like https://github.com/Safecast/reporting/blob/master/.ebextensions/grafana.config#L59-L66

Documenting some headway here:

• Set up github action CI pipeline
• Set up new docker-based dev env
• Run dev env on spot instances
• Use safecast_deploy for deployment
• Ensure CI pipeline can work for PR forks
• Move CI pipeline bundle step to https://github.com/Safecast/github-actions
• Try letsencrypt wildcard to avoid using LB in dev ($18/mo for elb)~
• Work out a nightly master grafana tag

[matschaffer]

Posting these images for https://forums.aws.amazon.com/thread.jspa?messageID=952947&#952947 since AWS forum images uploads seem to be broken at the moment.

[matschaffer]

So looks like the extra ASG got created with the stack then failed to get deleted (possibly because it was still coming online).

I think for the time being I'll plan to run these with just one instance type to avoid the weirdness.

[matschaffer]

I also managed to get a letsencrypt cert with this

PROD=https://acme-v02.api.letsencrypt.org/directory
STAGING=https://acme-staging-v02.api.letsencrypt.org/directory

docker run -it --rm --name certbot \
  -v "/etc/letsencrypt:/etc/letsencrypt" \
  -v "/var/lib/letsencrypt:/var/lib/letsencrypt" \
  certbot/certbot \
    certonly --manual \
    --preferred-challenges=dns \
    --email mat@safecast.org \
    --server "${PROD}" \
    --agree-tos \
    --manual-public-ip-logging-ok \
    -d "*.safecast.org,*.safecast.cc"

It was pretty simple, but it looks like renewal will require updating the TXT record for the domains and I'm fairly certain mediatemple won't let us do that, so we'd need someone to update mediatemple DNS manually every 2.5 months and run the renewal which sounds like a recipe for a broken site for anything under the safecast.org domain. Probably not worth the $100/mo we spend on LBs.

So it might make sense to have the dev envs run on safecast.cc with letsencrypt at some point. I'll look at setting reporting up that way anyway and we can evaluate what we want to do with other apps.

[matschaffer]

Looks like https://github.com/go-acme/lego might be a better option for the certs since it has route53 support directly. Will give it a shot.

[matschaffer]

Looks like beanstalk supports shared ALBs now so gonna abandon the letsencrypt & manual cert management. I think we can just have one prd and one dev ALB for all apps and get the price down plenty.

[matschaffer]

I just noticed https://hub.docker.com/r/grafana/grafana-dev/tags?page=1&ordering=last_updated has some sort of "nightly" images, but not sure how up to date they're kept. The tags are all 7.5.0 but the latest release is 7.5.1

[matschaffer]

AWS had our old instance slated for replacement so I went ahead with this as-is. We still don't have a good answer to running grafana nightly builds (since they don't provide docker images for them), but we also can't upgrade to 8 at present anyway since the panodata map plugin doesn't run on it.

I merged Safecast/reporting#11, created a new prod env, so I'd say we can call this done.