Repository navigation
How to run kubernetes e2e tests with kind and upgrade the cluster components with your local changes #1181
Description
Activity
More complete instructions here
This is how I do it :
-
Work in your kubernetes branch, preferably in the default go src location
-
Create your kind node image:
kind build node-image -
Create your kind e2e cluster config kind-config-yaml:
# necessary for conformance kind: Cluster apiVersion: kind.x-k8s.io/v1alpha4 networking: ipFamily: ipv4 nodes: # the control plane node - role: control-plane - role: worker - role: worker-
Set your KUBECONFIG env variable (KIND generates the conf based on it):
export KUBECONFIG="${HOME}/.kube/kind-test-config" -
Use the previous config to create your cluster:
kind create cluster --config kind-config.yaml --image kindest/node:latest -v4 -
Create your e2e Kubernetes binary (from your Kubernetes src code):
make WHAT="test/e2e/e2e.test" -
Execute your tests:
./_output/bin/e2e.test -context kind-kind -ginkgo.focus="\[sig-network\].*Conformance" -num-nodes 2
Note: updated with new KIND features and known issues
Reacted by Davanum Srinivas, 9r0k, Jason Simmons, Zack Zeng, Surya Seetharaman, Amim Knabben, Matthias Bertschy, Mustafa Elbehery, Vedant Shrotria, jony montana and 15 moreReacted by Daman Arora-
To expand on this and for completenes, let's say that you have a change in kube-proxy, you don't have to redo all the steps, you just need to:
- build the new image:
bazel build //build:docker-artifacts- load it in your current kind cluster:
kind load image-archive bazel-bin/build/kube-proxy.tar- check that it has been loaded:
docker exec -it kind-control-plane crictl images IMAGE TAG IMAGE ID SIZE docker.io/kindest/kindnetd 0.5.4 2186a1a396deb 113MB docker.io/rancher/local-path-provisioner v0.0.11 9d12f9848b99f 36.5MB k8s.gcr.io/coredns 1.6.5 70f311871ae12 41.7MB k8s.gcr.io/debian-base v2.0.0 9bd6154724425 53.9MB k8s.gcr.io/etcd 3.4.3-0 303ce5db0e90d 290MB k8s.gcr.io/kube-apiserver v1.17.0 4a0e3a87a5e22 144MB k8s.gcr.io/kube-controller-manager v1.17.0 fa313a582b872 131MB k8s.gcr.io/kube-proxy-amd64 v1.18.0-alpha.1.633_8428af6fd79e1f-dirty 5dd06b2bb1290 124MB k8s.gcr.io/kube-proxy v1.17.0 e3dd0e2bea53a 132MB k8s.gcr.io/kube-scheduler v1.17.0 cb8feb1d83dd3 112MB k8s.gcr.io/pause 3.1 da86e6ba6ca19 746kB- and modify the kubernetes deployment to pick the new image, in this case you can follow these instructions to switch the container image of the daemonset
https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/
To expand on this and for completenes, let's say that you have a change in kube-proxy, you don't have to redo all the steps, you just need to:
for that kind of power user, you can do this today and write yourself a small script.
at the moment I'm specifically focused on #986 which is about mimicing the kubernetes flow.
if you're building with bazel you already no-op most of the build if you didn't change the other binaries, I wouldn't recommend taking shortcuts vs just running a node image build.
Reacted by Antonio Ojea and Mustafa Elbeheryyeah, you are absolutely right.
I'm just sharing some experiences, specifically, this last case I've added is useful when you are doing some exploratory analysis on a complex setup and/or your prototyping something.- You probably need to add some flags. Please share your test invocation.…On Fri, Jan 31, 2020, 00:31 9r0k ***@***.***> wrote: @aojea <https://github.com/aojea> Yes, it has run the test suite and show the results, but the process is not end itself, after tests, it blocked with INFO: Running AfterSuite actions on all nodes — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1181?email_source=notifications&email_token=AAHADK6UGUUR2OTTBUFEKTDRAPOWFA5CNFSM4J4JT3Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKN5ASI#issuecomment-580636745>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK544XT77SYYKCCIJQDRAPOWFANCNFSM4J4JT3YQ> .
@BenTheElder I just use the
bazel-bin/test/e2e/e2e.test -context kind-kind -ginkgo.focus="\[sig-network\].*Conformance" -num-nodes 2 -ginkgo.parallel.total 8naive question, do you have at least 2 workers in your cluster?
naive question, do you have at least 2 workers in your cluster?
Yes, i follow your steps. The test can be run, but it does not end by itself
Reacted by Antonio OjeaI'm not certain about that invocation, currently we use hack/ginkgo-e2e.sh in k8s which handles a few more things, you can check our own hack/ci dir for the current scripts we use to do this, they set some other things
Reacted by Antonio Ojeain parallel, I'll check if I have some additional environment variables that are missing causing this behavior.
20 remaining items
the solution is to remove the -ginkgo.parallel.total 8 , there are more issues referring to this behavior
What is the correct way, or best practice, to run tests in parallel?
I have got excellent advice from @aojea in the past (thanks!) that I am still using so, I have done:
#go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo # once e2e=$GOPATH/src/k8s.io/kubernetes/_output/bin/e2e.test FOCUS="\[sig-network\].*Conformance" ginkgo --nodes=16 --focus="${FOCUS}" $e2e -- --num-nodes=2 --provider=skeletonand got execution time down by ~7 times (>500s -> ~75s).
Man, you seem to have a gist for everything 😄 Thanks. I think it might be the same advise I got years back
Reacted by Antonio OjeaThis information is documented somewhere?
This is a bit of a hack, but the gist linked above is the approach.
Otherwise the general full approach is to 1)
kind build node-imageand then create the cluster with that image, and then run the tests as normal like any conformant cluster, we also have scripts for that in this repo, and Kubernetes contributors can see them running on their PRs (e.g.pull-kubernetes-e2e-kind, see the logs which log exactly what is run).Reacted by Yuiko Mouri/close
@aojea: Closing this issue.
Details
In response to this:
/close
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
What would you like to be documented:
How to use kind to test your code with e2e tests
Why is this needed:
I know that Ben has this in the backlog, that there are several nice tutorials and videos, there is also a script in the repo to execute the e2e tests, ..., but I always fall back to my snippet :-)