About testing

[ccremer]

Starting situation

We have currently scaffolding for 3 different kinds of tests:

• Unit tests (plain old Go tests)
• Integration tests (starts an in-memory-Kubernetes-API server before running tests)
• end-to-end (e2e) tests (uses Kubernetes-in-Docker and Kustomize to fully deploy a K8up stack)

We are currently testing K8up with 3 different test frameworks:

• Go testing package (easy, fast, integrated into IDEs)
• Ginkgo (BDD). This was set up as part of the Operator-SDK rewrite. It includes the tools of starting the in-memory-API server, needs to be started from Makefile currently, limited debugging available)
• DETIK (bats-based) to run test cases with bash, so it's closer to actual user experience. Comes with Kubernetes assertion/helper tools.

But, we barely have any actual test cases yet. Most was tested somehow manually until recently. Or only recently we started adding tests to our PRs.

Questions

1. Do we keep using Ginkgo? Or should we rewrite to plain old Go testing tools? Some IDEs (e.g. Goland) do not have first class support for running Ginkgo-based tests.
2. How do we define which cases are to be covered?
3. What kind of metrics can/should we use to measure quality?
4. Style of tests: Table-driven tests, verbosity vs readability
5. In End-to-End testing framework research #156 we decided to separate testing K8up from testing (w)Restic. However, I would like to question this decision:

• When we start to document tutorials for the end user, we might as well "automate" the tutorial. Having an e2e test that covers the Happy Path of a tutorial with the exact same steps and manifests in the tutorial ensures that for whatever change incoming, our tutorial stays up-to-date.
• For testing the tutorial, we need an actual backup repository. This could be a sponsered S3 backup repo in the cloud, or locally with minio for example.

[ccremer]

Me, @cimnine and @Kidswiss had a lengthy discussion around testing and come to the following conclusions:

No, we try to get rid of Ginkgo in favor of better IDE support. The value Ginkgo provides resp. the operator SDK scaffolds, is not worth the addition of a separate framework compared to plain Go's testing package.

We keep a distinction between Unit and Integration tests. However, it is by design or by test requirements whether a in-memory Kube API server is required for a test.

loose definition of Unit test: They're small, fast, isolated. In practice they don't need fakeclients or a K8s API server, best suited for small functions. A design that favors those kind of functions and tests should be preferred.

loose definition of Integration test: They're bigger in scope. Some tests (but not all) need K8s interaction, in which cases the K8s API server can be used, since there's a startup overhead that makes tests slow. We will try to avoid the K8s fakeclient and use in-memory K8s. Reason being that the fakeclient may not show up side effects which could give a false sense of security when tests pass, but not with a "real" API server (reconcile loops, race conditions etc).

We will provide short, but generic enough instructions on how to setup an IDE with working in-memory K8s so that those tests can be run and debugged without the need of Makefile.

[Kidswiss]

We will provide short, but generic enough instructions on how to setup an IDE with working in-memory K8s so that those tests can be run and debugged without the need of Makefile.

We could provide example configs for goland and vscode.

[Kidswiss]

Maybe we could provide some example use case for unit and integration tests. Something like:

Unit test: if you write a function you write a unit test that will test the happy paths, errors and edge cases of that function.

Integration test: you write a logical feature and ensure that it works. Example: ensure that the operator reads and merges global settings correctly.

[ccremer]

We could provide example configs for goland and vscode.

Where would you document such examples? should there be a dedicated DEVELOPING.md file in the repository, linked from the README?
Or should we make a section in the Antora documentation like "K8up for developers and contributors" where we get the full features of asciidoc/Antora, can structure it better and have everything in one place?
@tobru your thoughts?

Maybe we could provide some example use case for unit and integration tests. Something like:

Yes, providing a reference to "good" unit/integration tests helps understanding the "definition"

[tobru]

IMHO this belongs into the Antora docs

[ccremer]

We try to test as much as possible with Unit and Integration tests. But at some point we require confidence whether the actual, complete stack works on an actual cluster. Thus, on the level of e2e we test only the Happy Path.

We orient ourselves around the documentation. Those are the cases that are ideally e2e-tested. This should give enough confidence for a stable K8up.

In short, we use e2e test specifically to cover the use cases from the user's perspective. those tests should cover most of the how-tos and tutorials, directly or indirectly.

For the moment, we distinguish between e2e-testing K8up and e2e-testing the Tutorial. Because e2e testing the Tutorial (install Minio, mariadb and Wordpress requires effort that is not needed to verify that K8up works. To verify that K8up works, one might just create a dummy file and restore that, no need for mariadb and wordpress, as an example.

For every bugfix (related to code), there should be a new test case that proves the bug is permanently fixed. The level of test can go up to new e2e test if necessary.

[Kidswiss]

For every bugfix (related to code), there should be a new test case that proves the bug is permanently fixed. The level of test can go up to new e2e test if necessary.

This could be tricky to determine though.

[ccremer]

Maybe saying "addressing the bug" is worded a bit better than proving. It's about avoiding a regression or a duplicate issue because the same bug has later resurfaced again after some changes.

[ccremer]

We delayed this question. CodeClimate's code coverage and maintainability metrics are good enough for now.
However, we are aware that CodeClimate has reported a lot of issues that should be eventually fixed. Some code though is known to be ugly, so we try to fix the most painful ones, while ignoring some minor issues and keeping them in mind for an eventual K8up v2. We don't know when we can start developing K8up v2.

[ccremer]

The current table-driven test design, structure and naming scheme is acceptable. As a reference example, see here: https://github.com/vshn/k8up/blob/v1.0.0-rc2/prebackup/prebackup_test.go

[ccremer]

see question 2) : We will first start with an e2e test that covers K8up backup/restore feature, then we discuss whether more effort is justified for testing the tutorial.

[Kidswiss]

You're right for e2e tests we can't avoid to also test wrestic. But wrestic should be completely out of scope for integration tests.

[cimnine]

I'd argue that we don't need to test wrestic, but it should be part of the tests. Like we won't test the K8s controller or minio, but they will be part of an e2e test, too.

[Kidswiss]

True, testing wrestic should be in wrestic's scope.

[ccremer]

Next steps: create issues that bring most value when testing.

• Add e2e tests that covers: Minio installation with Helm (Adapt tutorial if needed so it matches also Helm method), run K8up backup, restore from Backup (see Add e2e tests that creates a backup and restores from it #272)
• Open PR to get rid of Ginkgo, and enable integration test in GitHub PR workflows (atm only unit tests are run). Also include instructions to set up local development for integration tests.
• Add e2e tests that verifies scheduling of prebackup pods. Especially interesting is also the case of existing prebackup deployments or backup CR objects, though this could also be done in an integration test maybe (up to the author). (see Add e2e tests to test PreBackupPods #273)
• Open PR to document the findings and definitions of this discussion in Antora (see Document testing guide for developers #271)

Unit tests can be added organically whenever some functions are changed and easy testable. Though the biggest bang for the buck are currently e2e tests

[Kidswiss]

Unit tests can be added organically whenever some functions are changed and easy testable. Though the biggest bang for the buck are currently e2e tests

We could add a bullet point to the PR template like:
"Unit tests have been written for the touched functions" or something like this.

[ccremer]

Hm, we already have such a check point., just worded differently. I think it should be up to the code reviewers to accept or demand more/different tests.