Basic end to end test

[zimnx]

Currently we don't have any automatic test in this repo which would validate even basic scenario such as spawning 1 rack with 3 nodes cluster.

For a start, we should have a test which spawns ScyllaCluster on local minikube.
Once we have baseline, all new features should come together with tests.

[espindola]

It would be really nice if the basic tests were not integration tests. Since the only thing an operator does is talk http, it is trivial to mock the other side.

We should still have integration tests, not only with Minikube but with any implementation we support (amazon, google, etc), but with good local tests we don't need to run that during the regular development cycle.

[yanniszark]

@zimnx @espindola may I recommend taking a look at k3d?
https://github.com/rancher/k3d

It's super-lightweight, works well with GitHub workflows and allows for multi-node clusters.
For instance, we use it in our oidc-authservice repo with great success. I've written a small blog post about it in case you are interested:
https://www.arrikto.com/devops/k3d-github-actions-kubernetes-e2e-testing-made-easy/

The test is very lightweight and runs in about 3 minutes (sets up k8s cluster, applies YAML, waits and performs checks).
I think it would also work great for testing the Scylla operator end-to-end, without slowing down development speed.

[espindola]

Yannis Zarkadas <notifications@github.com> writes:

@zimnx @espindola may I recommend taking a look at k3d? https://github.com/rancher/k3d

If I understand it correctly, it is basically a lighter version of minikube? If so I don't see why not moving from minikube to it for both development and first pass at integration tests. I still think there is value in having dedicated "unit" tests by mocking the API server: * They are much faster. * A test failure points to a very specific location. It is something like: I was expecting a POST request where FOO was set to 42, but got 43. * It is possible to test error paths that cannot be easily tested with integration tests. As an example, I wrote an independent implementation of the sample controller. It is at https://github.com/espindola/sample-controller-from-scratch ``` $ go test -count=100 ./pkg/controller ok github.com/espindola/sample-controller/pkg/controller 0.245s $ go test ./pkg/controller -coverprofile cover.out -coverpkg ./pkg/controller,./pkg/kubeapi ok github.com/espindola/sample-controller/pkg/controller 0.012s coverage: 97.3% of statements in ./pkg/controller, ./pkg/kubeapi ``` Cheers, Rafael

[yanniszark]

If I understand it correctly, it is basically a lighter version of
minikube?

More or less. Plus, it supports multi-node setups, using multiple docker containers.

If so I don't see why not moving from minikube to it for both
development and first pass at integration tests.

I would recommend doing so.

I still think there is value in having dedicated "unit" tests by mocking
the API server:

Of course :) Both are needed (integration and e2e).
Let me share the approaches I have seen so far:

• Mock the API-Server and store resources in-memory:
  • Client-go: https://godoc.org/k8s.io/client-go/kubernetes/fake
  • Controller-Runtime: https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client/fake
• Run the API-Server / etcd binaries locally for unit testing:
  • Controller-runtime: https://godoc.org/sigs.k8s.io/controller-runtime/pkg/envtest

I don't think the approaches above can simulate errors, but you can probably achieve that if client is an interface (which it is).

As an example, I wrote an independent implementation of the sample
controller. It is at

This is great! You should do a blog post with that in the future, it would be very educational!

[espindola]

Yannis Zarkadas <notifications@github.com> writes:

> If I understand it correctly, it is basically a lighter version of > minikube? More or less. Plus, it supports multi-node setups, using multiple docker containers.

Minikube start has a --nodes option. It is not working?

> I still think there is value in having dedicated "unit" tests by mocking > the API server: Of course :) Both are needed (integration and e2e). Let me share the approaches I have seen so far: - Mock the API-Server and store resources locally: - Client-go: https://godoc.org/k8s.io/client-go/kubernetes/fake - Controller-Runtime: https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client/fake - Run the API-Server / etcd binaries locally for unit testing: - Controller-runtime: https://godoc.org/sigs.k8s.io/controller-runtime/pkg/envtest I don't think the approaches above can simulate errors, but you can probably achieve that if client is an interface ([which it is](https://godoc.org/sigs.k8s.io/controller-runtime/pkg/client)).

Semantically I think that client-go's fake is similar to what I did, but I did the mock at the http transport level. I will be the first to admit that the tests I have are a mess and need to be refactored, but I hope it is possible to get full coverage of the controller itself (not the low level APIs) without a complicated infrastructure.

> As an example, I wrote an independent implementation of the sample > controller. It is at This is great! You should do a blog post with that in the future, it would be very educational!

I am more of mailing list kind of developer, so I wrote https://groups.google.com/g/kubernetes-dev/c/2GT5T4MzwFk Cheers, Rafael