Explore testing strategies

[paulsilcock]

Currently code is linted and validated to check for basic errors, and we output the terraform plan on pull requests.

It would be nice to verify policies - e.g. check that only secret-manager google service account can access secrets.

Creating ephemeral environments for pull requests would give further assurances that the generated terraform plan will apply and behave correctly (terratest looks useful here). One approach is to create GCP projects on-the-fly, although this requires a Google Workspace account if we want to do this programmatically from terraform (which costs money!) We could use raw gcloud commands to do this instead. Alternatively we could create/destroy clusters inside the current project, but there is a potentially troublesome lack of isolation here.

Testing ingress/static IP addressing could also prove tricky, although it looks like Google Domains can be imported into GCP. In tandem with Cloud DNS, we could create DNS records for pull request environments, e.g. *.pr5.pauljs.io.