Epic: Automated Self-Hosted Testing

[corneliusludmann]

Summary

Set up werft jobs that automatically install and test (running the existing integration tests) Gitpod self-hosted from main on 4 representative infrastructure environments every night as well as manually on demand. Also the update path should be tested

Context

Before we can publish a new Gitpod self-hosted release, we need to test the installation on all supported platforms. Doing these by hand is annoying, error-prone, and time-consuming. That's why we need to automate this. This will build our confidence in our releases, which in turn will increase trust with users and customers.

Value

Regular, automated tests ensure more reliable Gitpod releases, increasing our own confidence and our customer's confidence.

Acceptance Criteria

• We have a werft job that tests the latest self-hosted from main and runs the existing integration tests.
• Test runs on 4 representative environments (tbc what these are, see below for first draft)
• Tests for fresh installations as well as updates from the previous version/release.
• Notifications about failed tests.

Measurement

• Number of tests run per week.
• Number of different environments we test on on a nightly basis.

Growth Area

• Onboarding

Proposed Test Environments

1. AWS
2. GKE
3. AWS - airgapped
4. AWS - self-signed certs

Milestones

• Milestone 1 - MVP with a basic werft pipeline to run tests
• Milestone 2 - Support all cloud providers
• Milestone 3 - Additional testing dimensions
• Milestone 4: Unittests for terraform script

Full testing surface area defined here (internal)

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[nandajavarma]

Based on the discussion I had with @corneliusludmann, I have made a very rough note on notion, about things we discussed during the chat and a small plan on where to start.

[lucasvaltl]

Closing this as we fulfilled the acceptance criteria - we are running nightly tests on 4 platforms :)