Add tests for external dependencies in GCP

[nandajavarma]

Description

This PR adds the external dependencies test for GCP cloud platform. These dependencies include cloudsql as external database, GCR as external registry, and cloud storage as external storage. These dependencies are randomly selected for each test providing us a overall testing of every combination over a course of days.

Related Issue(s)

Fixes #

How to test

You can run the following command to test one of the available combinations:

werft run github -j .werft/gke-installer-tests.yaml

Release Notes

NONE

Documentation

Werft options:

• /werft with-preview

[adrienthebo]

#10688 has been merged, this is ready to be rebased against main. 😄

[adrienthebo]

I see some potential for risk in how we're selecting randomized test cases but we should be able to merge this in time for the end of Q2. Will review this again after the rebase but I don't see any blockers.

[adrienthebo]

Question: Do we have a good understanding of our database load under normal operating conditions and test condition? Given that we're not going to be hitting the database that hard can we opt for a cheaper DB tier and increase it if testing scenarios demand more memory/throughput?

[nandajavarma]

This is a good point, ideally we can make this configurable and use a cheaper one for tests. I was basically just translating the reference architecture here. I was thinking that since our test setup is ephemeral the extra cost we pay here is quite small. May be I am wrong, and it adds up. I will make a note for this for a follow-up

[adrienthebo]

Suggestion: performing randomization in this location means that loading this file isn't an idempotent operation. If a randomized test configuration fails we'll likely need to repeat that test case (either to rule out a transient failure or reproduce the failure). Because these settings are encoded during the file evaluation we don't have a means of injecting an exact configuration.

Phase parameterization can be kept out of scope for this PR (espcially because it'll take time to define how we configure phases), but we should definitely follow up on this in another issue/PR for Q3. If we defer to Q3 I'd love to capture this requirement in a ticket before we merge this PR.

EDIT: Leaving the above comment as it's still relevant, but less so.

In install/tests/Makefile we do provide mechanisms for setting the exact configuration so we should be able to reproduce a given test case if we omit werft. This reduces the challenge of randomizing these values 🎉 , but we'll still need to account for this in the future.

[adrienthebo]

Note: will this subdomain need to change given the issues we've had with domain level wildcard letsencrypt issuance?

[nandajavarma]

We will highly likely stick with the same subdomain, just we will try and re-use the certs issued across multiple sub-domains(max 200 subdomains can be added under a certificate, iirc). In reality, I don't think we will hit a limit even if we create new certs per domain. This week it so happened because I was running too many tests on my own

[adrienthebo]

LGTM!