Discuss re-structuring tests, adding envtest and introducing a guide for new tests

[pmalek]

Problem statement

Currently we have 3 levels of tests: unit (with some usage of envtest already), integration tests, and e2e tests.

This issue tracks the effort to

• discuss re-structuring the tests in a way that's more organized
• discuss and document adding dedicated envtest based tests (e.g. in test/envtest/)
• add a guide helping developers too choose which type of tests to write when adding new ones Guide for developers: how to test KIC #4107

Why Anything?

• some integration tests could be on a lower level - this would make them faster
• the envtest approach is promising at achieving the above

[shaneutt]

I agree that the testing landscape can be a little bit confusing, so I think this effort is probably quite helpful for future contributors and maintainers. More envtest based coverage is compelling because it would enable us to exercise more controller code at something a bit closer to a unit test level, just a matter of putting in the time. Also just a reminder that we have a # Testing section in our CONTRIBUTING.md which might be a good place to expand, or at the very least just remember that it's there when we work on new documentation.

[programmer04]

Food for thought

Maybe envtest should be part of KTF.

If it looks like a duck, swims like a duck, and quacks like a duck, then it probably is a duck.

The above is not fully true in this case, because created pods, etc. don't do anything useful. But on the other hand, all K8s clients, etc. work with such an environment as with a normal cluster. Having the same interface as for Kind cluster for such envtest env would be handy in my opinion. Wrapping creation and tear down envtest in familiar interface.

[pmalek]

That could definitely happen at some point.

One note for this that I have is that for now, envtest related code is getting shape (in pain and slowly). I believe we'll get much more speed by extending this in-repo rather than in ktf. We can decide if we'd like to move it when we have the structure of its API in place.

One more thing that got into my head is that users might be surprised when using it given the duck analogy that you brought up. envtest is a very limited environment, you don't get any upstream controllers, you cannot delete a namespace and more.

If we end up needed envtest in KGO, then surely it will make sense but we'd have to rethink the API as currently, the API relies on having "a cluster" where envtest environment doesn't really constitute a cluster in a way.

[programmer04]

I agree, we can iterate faster without it being in KTF, and we can work on making it more convenient directly in KIC repo.

I think about a more streamlined API for envtest, in my opinion such environment could implement this portion (+ DumpDiagnostics) of the Cluster interface.

[pmalek]

I think about a more streamlined API for envtest, in my opinion such environment could implement this portion (+ DumpDiagnostics) of the Cluster interface.

In this case we'd need to come up with a structured API re-design, extracting said part of an interface, giving it a proper and meaningful name etc