Add tests to make sure services are healthy

[daniel-sanche]

We want to make sure fresh sandboxes don't have any errors. Either error logs, or failing SLOs

[daniel-sanche]

Hey @winter-code, since this is your first bug, I'll leave some set-up instructions here:

Background References

• Kubernetes 101
• Cloud Ops Sandbox User Guide
• GitHub Actions Documentation
• GCP Python Logging Client Library Guide

Deployment

• see CONTRIBUTING.md
• TL;DR:
  • clone the repo's develop branch (we use separate branches for development and stable release code. more info on our process here)
  • create a GKE cluster with make cluster PROJECT_ID=my-project (or other method. Contributing file describes alternatives, but the makefile should be simplest)
  • build/deploy the local code to your cluster using make deploy PROJECT_ID=my-project

Viewing Logs Looking for Errors

• Look through the logs coming from your cluster in the Cloud Logging UI
  • You can use the GKE Workloads view to check the logs for each service individually
• Look through all the services and see if you see any orange or red error logs. If not, we can skip to writing tests. If you do, ping me and update this bug, and we can try to track down the cause

Writing Tests

• we want to add tests to make sure services will stay healthy going forward. I had mentioned adding a test to the GitHub Actions file, but actually, we can probably add it to the existing provisioning test file (which is already triggered as part of the GitHub Actions workflow)
• As part of TestGKECluster, you can add a new function that uses the GCP python logging client library to read logs from the cluster, filtered by severity.
• If you see any with severity of WARNING or higher, fail the test
• you may also want to wait until X number of logs come in, to make sure you are getting a good sample of the running service

[daniel-sanche]

Also, the previous comment was focused on the GKE code, but we also have a new service running on App Engine. So after the GKE part, you'll have to do the same deployment, viewing logs, and writing tests steps against that one too. But we can do that part at the end

[daniel-sanche]

This is partially addressed in #932, although we may want to check for error logs. (It currently just checks to make sure logs are coming in)

[minherz]

Correctness of Online Boutique is controlled via the pinned version in kustomization configuration.