Epic: Expand Self-Hosted Automated Testing Dimensions

[lucasvaltl]

Summary

This is a follow up epic to #7316 to increase the dimensions we test on - i.e. the environments we test in, regarding which dependencies are used, different configurations etc. The documentation for the current state of automated test can be found here: internal.

Context

After having the basic framework for automated testing of self-hosted gitpod within different environments in place, we want to increase our test coverage regarding the environments we test on. This should allow us to further booster confidence for a larger set of customers. and reduce the time it takes for us to test manually.

Value

• Increased Confidence due to higher test coverage: Due to the bring-your-own "hardware" and dependencies nature of Self-Hosted, it will be running in many different environments and configurations. As such, we need to ensure that any changes we make to Gitpod run smoothly in a large part of these environments - at the very least on our reference architecture. Increasing the amount of configurations we test on will help with this
• The more we can test in an automated manner, the less we have to test manually. This means more time to focus on feature work.

Acceptance Criteria

• Tests run on all of our reference architectures, and all variations of them
• We test for at least 70% of the possible testing dimensions laid out in this (internal) document.
• Time to release a self-hosted release is reduced to half a day

Measurement

• We can map our test coverage to our internal list of possible testing dimensions in order to roughly see how large our coverage (regarding what we test on, not what we test) is.

Issues

Increase test coverage and ensure tests pass:

• Visualisation for automated (self-hosted) failed tests for other teams #11273
• Work with other to expand what they test #11274
• Self hosted automated tests with test customisation patch  #11275
• Randomize testing against stable branch and unstable branch #11679

Terraform Tooling:

• Example directories showing how to use self-hosted Terraform modules #11279
• Create issues with tasks required to align terraform to ref arch for all existing cloud providers #11280
  • Sync AWS terraform module with reference architecture #11533
  • Sync Azure terraform module with reference architecture #11534
  • Sync GCP terraform module with reference architecture #11535
• Research ways to maintain the reference architecture and terraform modules in Sync, and easy way to get to one from the other #11281
• Validate and lint terraform modules #11029

Increase infra coverage:

• Run automated self-hosted tests on all supported Kubernetes versions #11349
• Automated testing with S3 backed in-cluster registry #11269
• Automated testing with Self-signed certs #11270
• Automated testing using Airgapped bundle #11271
• Customise operating system images - test on specific ubuntu version #11272

Other issues:

• Fix certificate rotating with self-hosted automate testing to not run out of certs #11276
• Clean up old self-hosted preview environments #11277
• Implement basic cost management for automated testing cloud projects #11278
• Automate builds of self-hosted Gitpod for preview environments  #7824
• Automate pulling of AMI IDs for testing EKS #12402

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.