tests(e2e): introduce environment wrapper and test suites

[phisco]

Description of your changes

This PR introduces the following changes:

• introduces the concept of "Test Suite", see test/e2e/README.md for more details.
• switches the CI job from a matrix built on "areas" to achieve some kind of parallelism, to one built on "test suites" (areas could still be added later on as an additional parameter to the matrix to improve times)
• introduces a dedicated configuration struct, globally instantiated from the e2e package as E2EConfig
• WARNING: switch e2e-framework to use unreleased versions, mainly due to the fix to label selection introduced in kubernetes-sigs/e2e-framework@b38dd95
• adds a BeforeEachFeature checking that all features are defining the a label ascribing them to a specific test preset, to avoid having unselected features by mistake, this is due to the fact that e2e-framework doesn't modifying the feature from the provided function so there is no way to implicitly add a default value to all the features not having it set. However "explicit is better than implicit", so I feel it's better like this.
• sets the e2e matrix jobs not to fail fast, to avoid having successful jobs being killed due to a flaky test from another suite of tests, we should not have them, but it's inevitable and in such cases having the other successful runs helps identifying whether it's an actual issue (all red) or due to flakiness.

Test suites allow to define custom groups of tests alongside required additional conditional setup steps and custom Crossplane installation options which currently allow to run both all the non alpha feature tests against the default Crossplane configuration, but also to run for each alpha feature all the basic and feature specific tests against a Crossplane installation with the selected alpha feature enabled from the start. We run the uninstall and upgrade tests as part of the basic suite, so we are still covering the "what happens to a running Crossplane instance if we enable/disable this alpha feature?". This will allow us to gain in the future the confidence required to promote alpha features to beta, knowing that none of the basic tests is failing.

AFAICT, test suites should transparently work with all current test selection modes, e.g. -test.run <regexp> or e2e-framework selection via labels/features/assessments.

TODO:

• according to the output of Define a guideline around OCI image management used in e2e tests #4294 and Define a guideline around OCI image management used in e2e tests #4294, we could decide to also conditionally build e2e images based on test suites, defaulting to building them all if no preset is defined explicitly.
• add flags to allow configuring various hardcoded fields, see // TODO(phisco): make it configurable comments in the code.
• Add debug logs to the TestMain function to gain visibility on the actual runtime configuration.
• potentially add /e2e command to run e2e test suites on-demand from the PR, reworking the tests we run by default on each push to a PR.

I have:

• Read and followed Crossplane's contribution process.
• Added or updated unit and E2E tests for my change
• Run make reviewable to ensure this PR is ready for review.
• [] Added backport release-x.y labels to auto-backport this PR if necessary.

[phisco]

FYI: I wanted to open an issue first, but while trying to validate my idea on a long train trip, I ended up mostly implementing the whole thing and to change my initial idea according to a few limitations from the e2e-framework and go testing framework in general. So in the end I decided it would have been easier to let the code speak directly, but feel free to challenge or discuss the whole approach as if it was a proposal instead 🙏

[pedjak]

LGTM

[lsviben]

Nice improvements, thanks @phisco . Added one minor question

[phisco]

I would like to have a feedback from @negz too when he's back.

[negz]

A few questions, but I like the general direction here.

[negz]

I've noticed that some other users of e2e-framework have a sub package per "suite". Would that approach make sense for us? I think in that world all of these init functions would become part of TestMain.

[phisco]

I tried moving them to separate sub packages too in the beginning, but then you need to either inject the environment from the root to all the sub packages in some way or reinitialize it for each one of them or make it importable from the root package, which would mean moving it to a non _test.go file. I tried looking around before this iteration, but couldn't find applicable examples, we can definitely think about that for the next iteration.

[negz]

This looks great, thank you!