Can't run tests locally

[grantr]

When I run go test ./pkg/..., I get a bunch of errors that seem to be complaining about a missing /usr/local/bin/kubebuilder/bin/etcd executable:

fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory

So I try to run ./test.sh but that exits immediately:

$ ./test.sh
using tools

Are there any docs on how to run tests?

[DirectXMan12]

You'll need a certain set of components (kube-apiserver, kubectl, etc) to run the tests, and they either have to be in a certain location, or you have to set the TEST_ASSET_<thing> environment variables to point at the right place for those components. After that, running go test should just work.

I've got a PR up that makes some improvements to this process (#86), and will gladly attempt to make it easier

[grantr]

Probably the easiest way to make it easier is to document how to do it :)

Is there a way to just run unit tests locally and leave the e2e tests for CI?

[DirectXMan12]

The tests in controller-runtime are hybrid unit/integration tests -- most of them don't use mocks or stubs, and instead just use a real API server, but not the rest of the kubernetes cluster. It makes the tests a lot easier to read and write, and means that we're more likely to be testing real behavior...

... which is a really long way of saying "no", since running just the tests that didn't require an API server instance would skip almost all of the useful tests.

The ./test.sh script downloads those binaries for you, but it's really designed to be run from CI. I'll try and split it out further and document it.

In the mean time, if you already have a kubernetes release hanging around, just set TEST_ASSET_{KUBECTL,APISERVER,ETCD} to the path to each of those components, and you should be good to go

[mattmoor]

In knative/serving (not yet using any of this stuff) I've found that the table testing we're doing, which is based on the OpenShift Ingress controller's tests (thanks to @smarterclayton for the pointer) are fairly effective at testing the behaviors we are interested in.

Now there's probably plenty we haven't hit yet that you folks have, but I would expect that the set of tests that need such heavy machinery to be the exception not the rule. :(

[DirectXMan12]

So, IMO, the problem with reactor tests is that one of two things often happen -- either you end up testing the actions that occur, and not the desired state (which is often suboptimal when testing kubernetes, and makes it harder to refactor), or you end up replicating the API server with maps, and as you end up using more API server functionality, you end up with more and more code to mock out the API server. Occasionally, you end up with both.

If you just use an actual API server, you get that functionality without too much mess or extra work.

(also, note that we don't need a full cluster -- just an single etcd instance and API server, and that the test harness actually takes care of starting these for you).

[grantr]

Ultimately I was able to run tests locally on a different machine that had previously installed kubebuilder. Yes this should be documented but that's a different issue. 😄

[vhosakot]

I saw the same error when I ran make test in the workspace created using kubebuiler due to missing /usr/local/kubebuilder/bin/etcd file on the host:

go test ./api/... ./controllers/... -coverprofile cover.out
...
2019-05-22T16:25:21.837Z	ERROR	controller-runtime.test-env	unable to start the controlplane	{"tries": 0, "error": "fork/exec /usr/local/kubebuilder/bin/etcd: no such file or directory"}

The following steps resolved the error for me and then make test worked fine:

wget https://github.com/kubernetes-sigs/kubebuilder/releases/download/v2.0.0-alpha.1/kubebuilder_2.0.0-alpha.1_linux_amd64.tar.gz
tar -zxvf  kubebuilder_2.0.0-alpha.1_linux_amd64.tar.gz
sudo mv kubebuilder_2.0.0-alpha.1_linux_amd64 /usr/local/kubebuilder

$ find /usr/local/kubebuilder
/usr/local/kubebuilder
/usr/local/kubebuilder/bin
/usr/local/kubebuilder/bin/etcd
/usr/local/kubebuilder/bin/kube-apiserver
/usr/local/kubebuilder/bin/kubebuilder
/usr/local/kubebuilder/bin/kubectl

workspace_directory$ make test
go fmt ./...
go vet ./...
go test ./api/... ./controllers/... -coverprofile cover.out
ok  	redacted/api/v1alpha1	7.420s	coverage: 2.4% of statements
ok  	redacted/controllers	5.816s	coverage: 0.0% of statements