Our preference on defining multiple e2e tests

[tadayosi]

Our e2e testing is growing and we already have many e2e tests. We also see the time of e2e test runs in CI growing along with features we add to the codebase.

So what is our preferable style to define multiple flavours of e2e tests on the same target function? For example, we want to test the cli command kamel run in multiple scenarios such as running with different locations of sources, or with different combinations of options, etc.
https://github.com/apache/camel-k/blob/main/e2e/common/cli/run_test.go

There are two ways of defining those different scenarios (flavours):

1. Have multiple distinct TestRun_XXX() functions, with each running on a different namespace and integration platform installation.
2. Have a single TestRun() function and define multiple t.Run("xxx", ...) blocks inside WithNewTestNamespace(), which uses only a single namespace and platform throughout the test suite.

Right now, we seem to mix the two styles without any explicit policies. But possibly we should prefer one over another.

What I'm thinking now is that perhaps we should prefer the option 2 throughout the e2e testing by default. It's because we only create and destroy one namespace and platform for each collection of tests and it might make our e2e testing more efficient and fast. One potential drawback on the option 2 is that it might introduce hidden dependencies to each test in the same collection so we might need to be careful about leftovers in each test. But I naively think it's rare to happen.

WDYT?

[tadayosi]

Another drawback in the option 2 is that you cannot run a single test in the collection as you can in the option 1. In the option 1, you can run a single test by using go test ... -run TestRun_XXX option.

[squakez]

Hey @tadayosi I'm always using the 2nd approach. As you've pointed out correctly, installing the operator for every test is seldom necessary. I guess there may be special situations where we need to create a new installation, but I'd say we should prefer option 2 when we can. About the single local testing, it's true, it may be a bit bothering, but I use to comment the code I don't need to test as a quick way to launch a single test.

[tadayosi]

OK, most of the e2e tests follow the 2nd approach already. The remaining tests that don't follow it are rather exceptional.

Let's then use this issue to rewrite the remaining to align with the 2nd approach.