How to run kubernetes e2e tests with kind and upgrade the cluster components with your local changes

[aojea]

What would you like to be documented:

How to use kind to test your code with e2e tests

Why is this needed:

I know that Ben has this in the backlog, that there are several nice tutorials and videos, there is also a script in the repo to execute the e2e tests, ..., but I always fall back to my snippet :-)

[aojea]

More complete instructions here

https://github.com/kubernetes/community/blob/master/contributors/devel/sig-architecture/conformance-tests.md#running-conformance-tests

This is how I do it :

1. Work in your kubernetes branch, preferably in the default go src location

2. Create your kind node image:
   kind build node-image

3. Create your kind e2e cluster config kind-config-yaml:

# necessary for conformance
kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
networking:
  ipFamily: ipv4
nodes:
# the control plane node
- role: control-plane
- role: worker
- role: worker

4. Set your KUBECONFIG env variable (KIND generates the conf based on it):
   export KUBECONFIG="${HOME}/.kube/kind-test-config"

5. Use the previous config to create your cluster:
   kind create cluster --config kind-config.yaml --image kindest/node:latest -v4

6. Create your e2e Kubernetes binary (from your Kubernetes src code):
   make WHAT="test/e2e/e2e.test"

7. Execute your tests:
   ./_output/bin/e2e.test -context kind-kind -ginkgo.focus="\[sig-network\].*Conformance" -num-nodes 2

Note: updated with new KIND features and known issues

[BenTheElder]

@mauilion
also #986

[aojea]

To expand on this and for completenes, let's say that you have a change in kube-proxy, you don't have to redo all the steps, you just need to:

• build the new image:

bazel build //build:docker-artifacts

• load it in your current kind cluster:

kind load image-archive bazel-bin/build/kube-proxy.tar

• check that it has been loaded:

docker exec -it kind-control-plane crictl images
IMAGE                                      TAG                                        IMAGE ID            SIZE
docker.io/kindest/kindnetd                 0.5.4                                      2186a1a396deb       113MB
docker.io/rancher/local-path-provisioner   v0.0.11                                    9d12f9848b99f       36.5MB
k8s.gcr.io/coredns                         1.6.5                                      70f311871ae12       41.7MB
k8s.gcr.io/debian-base                     v2.0.0                                     9bd6154724425       53.9MB
k8s.gcr.io/etcd                            3.4.3-0                                    303ce5db0e90d       290MB
k8s.gcr.io/kube-apiserver                  v1.17.0                                    4a0e3a87a5e22       144MB
k8s.gcr.io/kube-controller-manager         v1.17.0                                    fa313a582b872       131MB
k8s.gcr.io/kube-proxy-amd64                v1.18.0-alpha.1.633_8428af6fd79e1f-dirty   5dd06b2bb1290       124MB
k8s.gcr.io/kube-proxy                      v1.17.0                                    e3dd0e2bea53a       132MB
k8s.gcr.io/kube-scheduler                  v1.17.0                                    cb8feb1d83dd3       112MB
k8s.gcr.io/pause                           3.1                                        da86e6ba6ca19       746kB

• and modify the kubernetes deployment to pick the new image, in this case you can follow these instructions to switch the container image of the daemonset

https://kubernetes.io/docs/tasks/manage-daemon/update-daemon-set/

[BenTheElder]

To expand on this and for completenes, let's say that you have a change in kube-proxy, you don't have to redo all the steps, you just need to:

for that kind of power user, you can do this today and write yourself a small script.

at the moment I'm specifically focused on #986 which is about mimicing the kubernetes flow.

if you're building with bazel you already no-op most of the build if you didn't change the other binaries, I wouldn't recommend taking shortcuts vs just running a node image build.

[aojea]

yeah, you are absolutely right.
I'm just sharing some experiences, specifically, this last case I've added is useful when you are doing some exploratory analysis on a complex setup and/or your prototyping something.

[Rand01ph]

@aojea Thank you for your share. I try your snippet, but the test blocked with INFO: Running AfterSuite actions on all nodes. It's all right ?

[aojea]

@aojea Thank you for your share. I try your snippet, but the test blocked with INFO: Running AfterSuite actions on all nodes. It's all right ?

nope, it has to run the test suite and show the results ... :/

@benthelder is working in improving the UX #986

[Rand01ph]

@aojea Yes, it has run the test suite and show the results, but the process is not end itself, after tests, it blocked with INFO: Running AfterSuite actions on all nodes

[BenTheElder]

You probably need to add some flags. Please share your test invocation.

…

On Fri, Jan 31, 2020, 00:31 9r0k ***@***.***> wrote: @aojea <https://github.com/aojea> Yes, it has run the test suite and show the results, but the process is not end itself, after tests, it blocked with INFO: Running AfterSuite actions on all nodes — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#1181?email_source=notifications&email_token=AAHADK6UGUUR2OTTBUFEKTDRAPOWFA5CNFSM4J4JT3Y2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEKN5ASI#issuecomment-580636745>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHADK544XT77SYYKCCIJQDRAPOWFANCNFSM4J4JT3YQ> .

[Rand01ph]

@BenTheElder I just use the bazel-bin/test/e2e/e2e.test -context kind-kind -ginkgo.focus="\[sig-network\].*Conformance" -num-nodes 2 -ginkgo.parallel.total 8

[aojea]

naive question, do you have at least 2 workers in your cluster?

[Rand01ph]

naive question, do you have at least 2 workers in your cluster?

Yes, i follow your steps. The test can be run, but it does not end by itself

[BenTheElder]

I'm not certain about that invocation, currently we use hack/ginkgo-e2e.sh in k8s which handles a few more things, you can check our own hack/ci dir for the current scripts we use to do this, they set some other things

[aojea]

in parallel, I'll check if I have some additional environment variables that are missing causing this behavior.

[aojea]

@Rand01ph I hit the same issue, it seems related to skewed versions between the node image and the e2e.test binary :/

Feb 10 14:26:40.322: INFO: e2e test version: v1.18.0-alpha.2.580+acd97b42f3acb0
Feb 10 14:26:40.323: INFO: kube-apiserver version: v1.17.0

the solution is to remove the -ginkgo.parallel.total 8 , there are more issues referring to this behavior

onsi/ginkgo#206

[BenTheElder]

that sounds like a bug in e2e.test ...

[RainbowMango]

Can we use kind to run conformance tests with a specific CCM(cloud controller manager)?

In that case, we need to custom some components, such as:

• set kube-apiserver/kube-controller-manager with flags --cloud-provider=external
• set kubelet with flags --cloud-provider=openstack or gcp
• start an extra CCM. (This can be done by running the CCM container in the kind cluster, right?)

@BenTheElder Can you share your comments? I'm just starting to use kind, after went through the relative document, I'm not sure if kind support this case.

[MHBauer]

Since this has such a great title, can we share some of the links to the other tutorials and videos that exist?

[BenTheElder]

@MHBauer check out https://kind.sigs.k8s.io/docs/user/resources/

@RainbowMango maybe, CCMs have their own requirements, it's not something we're doing currently. I'm aware of some third party work in the past with the vsphere CCM and a vsphere fake + kind

sorry the integrated test work has gone so slowly, we keep having to paper over the existing tests, right now we're dealing with multiple critical bugs before daring to change the test workflow.

[MHBauer]

Beautiful page, thank you!

…

On 6/1/20 14:04:38, Benjamin Elder wrote: @MHBauer <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_MHBauer&d=DwMCaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=gSVznMk4DixLVQ8USuNSH0RuT_4psswDPWAf-45M48c&m=kVIkmj7zkl8fUzcD0aNTnxT-PWIIc7UkREQvJXhG2w4&s=0wP84bSg0JzTvrApqf0VXfQdvu4au1lvgercWBCxOfE&e=> check out https://kind.sigs.k8s.io/docs/user/resources/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__kind.sigs.k8s.io_docs_user_resources_&d=DwMCaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=gSVznMk4DixLVQ8USuNSH0RuT_4psswDPWAf-45M48c&m=kVIkmj7zkl8fUzcD0aNTnxT-PWIIc7UkREQvJXhG2w4&s=hqG9I3-fadEmtWsl3sEkbeVepFXV5NcA5xJ-_Y8WUCU&e=> @RainbowMango <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_RainbowMango&d=DwMCaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=gSVznMk4DixLVQ8USuNSH0RuT_4psswDPWAf-45M48c&m=kVIkmj7zkl8fUzcD0aNTnxT-PWIIc7UkREQvJXhG2w4&s=60xkxEAXy_qeK17Pph3G22W7f65lZZrEq6GjqEqUWJM&e=> maybe, CCMs have their own requirements, it's not something we're doing currently. I'm aware of some third party work in the past with the vsphere CCM and a vsphere fake + kind sorry the integrated test work has gone so slowly, we keep having to paper over the existing tests, right now we're dealing with multiple critical bugs before daring to change the test workflow. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_kubernetes-2Dsigs_kind_issues_1181-23issuecomment-2D637100891&d=DwMCaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=gSVznMk4DixLVQ8USuNSH0RuT_4psswDPWAf-45M48c&m=kVIkmj7zkl8fUzcD0aNTnxT-PWIIc7UkREQvJXhG2w4&s=pgR0w2fHeabE2ms9YMPaYDs-rw_JucM6cZJR1tH-lyc&e=>, or unsubscribe <https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADFYGAKBEPP7AC2ATM54CMDRUQJWNANCNFSM4J4JT3YQ&d=DwMCaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=gSVznMk4DixLVQ8USuNSH0RuT_4psswDPWAf-45M48c&m=kVIkmj7zkl8fUzcD0aNTnxT-PWIIc7UkREQvJXhG2w4&s=uTu4ulplGozSTCVOMM0MKr51Vvgth3_olul8k1Akj-E&e=>.

[aojea]

@BenTheElder I've created a script to upgrade the cluster components with the kubernetes repo local changes

https://gist.github.com/aojea/2c94034f8e86d08842e5916231eb3fe1

I wanted to add them to kind using a kind upgrade <component> <node> command or something similar, but I think that you have already something in mind so I prefer to sync with you first.

I'm sure this will boost development productivity

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[BenTheElder]

I think maybe we should just add a user guide page for this and it can eventually reference the shinier guide we want later. Even just a basic pointer to what the shell scripts we have now do / what CI is doing is probably worth getting a rough page up for sooner rather than later.

[uablrek]

the solution is to remove the -ginkgo.parallel.total 8 , there are more issues referring to this behavior

What is the correct way, or best practice, to run tests in parallel?

I have got excellent advice from @aojea in the past (thanks!) that I am still using so, I have done:

#go install -mod=mod github.com/onsi/ginkgo/v2/ginkgo  # once
e2e=$GOPATH/src/k8s.io/kubernetes/_output/bin/e2e.test
FOCUS="\[sig-network\].*Conformance"
ginkgo --nodes=16 --focus="${FOCUS}" $e2e -- --num-nodes=2 --provider=skeleton

and got execution time down by ~7 times (>500s -> ~75s).

[aojea]

https://gist.github.com/aojea/097b5a8418fbbcb2b55e72a4cf6e62f7

[uablrek]

Man, you seem to have a gist for everything 😄 Thanks. I think it might be the same advise I got years back

[YuikoTakada]

This information is documented somewhere?

[BenTheElder]

This is a bit of a hack, but the gist linked above is the approach.

Otherwise the general full approach is to 1) kind build node-image and then create the cluster with that image, and then run the tests as normal like any conformant cluster, we also have scripts for that in this repo, and Kubernetes contributors can see them running on their PRs (e.g. pull-kubernetes-e2e-kind, see the logs which log exactly what is run).