✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions #3652

randomvariable · 2020-09-16T16:03:00Z

Breaking out of #3593

Mostly upstreaming from CAPA.

The kubetest package allows a consumer to run kubetest (i.e. Kubernetes conformance) against
given clusters.

The kubernetesversions package provides helpers to determine the latest CI release of Kubernetes
and adds debian/ubuntu injection script kustomizations for based on prior work by @dims.

ginkgoextensions are common helpers that were added to CAPA, and are also present in the test/e2e
package.

Additionally add helpers to log running commands as well as gather JUnit files in a way that
is compatible with Prow.

Future PRs will add consumption of this in an e2e test and make CAPA consume these.

APIDiff will fail this PR, because a GetRESTConfig function has been added to the cluster proxy interface. This is used so that a client-go discovery client can be used to discover the API server's kubernetes version. Believe this has minimal impact due to no third-party implementations of the interface.

Signed-off-by: Naadir Jeewa jeewan@vmware.com

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Part of #3569
Part of #2826

randomvariable · 2020-09-16T16:26:16Z

/retest

vincepri

I can take a deeper look later

/assign @sedefsavas
for another set of 👀

test/framework/kubernetesversions/zz_generated.bindata.go

test/framework/kubernetesversions/template.go

test/framework/kubernetesversions/versions.go

test/framework/kubetest/setup.go

test/framework/kubernetesversions/bindata.go

test/framework/kubetest/run.go

randomvariable · 2020-09-17T12:01:39Z

/test pull-cluster-api-test

randomvariable · 2020-09-17T13:17:25Z

/test pull-cluster-api-test

sedefsavas · 2020-09-17T16:06:01Z

/lgtm

benmoss · 2020-09-23T13:30:39Z

/lgtm

benmoss · 2020-09-23T13:34:58Z

/test pull-cluster-api-verify-external-links

just want to see if this works 😄

fabriziopandini

@randomvariable thanks for this PR and sorry for lagging behind

This PR looks much more cleaner than the original one, and I'm looking forward to see this used by different providers
My only concern is about using this in docker, because in my experience there could be potential resource bottlenecks when executing the install scripts on several containers running on the same machines, and this can lead to flakes. But let's get this merged so we can test this in practice.

fabriziopandini · 2020-09-24T09:10:15Z

test/framework/kubernetesversions/template.go

+	// KubeadmConfigTemplateName is the name of the KubeadmConfigTemplate resource
+	// that needs to have the Debian install script injected. Defaults to "${CLUSTER_NAME}-md-0".
+	KubeadmConfigTemplateName string
+	// KubeadmControlPlaneName is the name of the KubeadmControlPlane resource
+	// that needs to have the Debian install script injected. Defaults to "${CLUSTER_NAME}-control-plane".
+	KubeadmControlPlaneName string


Should we add Machine pool as well?

We're only customising the PreKubeadmCommands and Files bits, so we shouldn't need to touch MachinePools do we?

if this is the case, why do we need KubeadmConfigTemplateName and KubeadmControlPlaneName?

those are the resources that get customised.

Please look at https://github.com/kubernetes-sigs/cluster-api/pull/3652/files#diff-646b9b4cde9720e1732b0305f6f337edR174-R190

I just saw this, @randomvariable @fabriziopandini we might need MachinePools as well because they use KubeadmConfig not KubeadmConfigTemplate, see how we did it in Azure: https://github.com/kubernetes-sigs/cluster-api-provider-azure/blob/5f7473fd8221248ca01e35365e4e862a27bf5acf/templates/test/prow-machine-pool-ci-version/patches/machine-pool-ci-version.yaml

Ah, the KubeadmConfig for a MachinePool. Makes sense. Will PR.

test/framework/log/log.go

randomvariable · 2020-09-24T09:45:38Z

My only concern is about using this in docker, because in my experience there could be potential resource bottlenecks when executing the install scripts on several containers running on the same machines

Same, the other way would be Sonobuoy, but unfortunately we can't simultaneously import Sonobuoy and clusterctl framework because Sonobuoy imports clashing versions of API machinery. We can probably be revisit when we bump controller runtime in v1alpha4.

…nd additional suite_helpers Mostly upstreaming from CAPA. The kubetest package allows a consumer to run kubetest (i.e. Kubernetes conformance) against given clusters. The kubernetesversions package provides helpers to determine the latest CI release of Kubernetes and adds debian/ubuntu injection script kustomizations for based on prior work by dims. ginkgoextensions are common helpers that were added to CAPA, and are also present in the test/e2e package. Additionally add helpers to log running commands as well as gather JUnit files in a way that is compatible with Prow. Future PRs will add consumption of this in an e2e test and make CAPA consume these. Signed-off-by: Naadir Jeewa <jeewan@vmware.com>

k8s-ci-robot · 2020-09-24T10:02:48Z

@randomvariable: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
pull-cluster-api-verify-external-links	`995578c`	link	`/test pull-cluster-api-verify-external-links`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

vincepri · 2020-09-24T19:59:09Z

/milestone v0.3.10

randomvariable · 2020-09-25T12:11:18Z

Any update on approval?

sedefsavas · 2020-09-25T16:01:43Z

/lgtm

vincepri · 2020-09-25T18:01:56Z

/approve

k8s-ci-robot · 2020-09-25T18:02:20Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

vincepri · 2020-09-25T18:32:28Z

/override pull-cluster-api-apidiff
/override pull-cluster-api-verify-external-links

k8s-ci-robot · 2020-09-25T18:32:32Z

@vincepri: Overrode contexts on behalf of vincepri: pull-cluster-api-apidiff, pull-cluster-api-verify-external-links

In response to this:

/override pull-cluster-api-apidiff
/override pull-cluster-api-verify-external-links

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

k8s-ci-robot added do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Sep 16, 2020

k8s-ci-robot requested review from detiber and JoelSpeed September 16, 2020 16:03

k8s-ci-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Sep 16, 2020

randomvariable force-pushed the test-framework-extensions branch from b60e0bd to 95e8053 Compare September 16, 2020 16:03

randomvariable changed the title ~~test framework: Add kubetest, kubernetesversions, and ginkgo extensions~~ test framework: Add kubetest, kubernetesversions, log and ginkgo extensions Sep 16, 2020

randomvariable force-pushed the test-framework-extensions branch from 95e8053 to 72953d0 Compare September 16, 2020 16:06

k8s-ci-robot removed the do-not-merge/invalid-commit-message Indicates that a PR should not merge because it has an invalid commit message. label Sep 16, 2020

randomvariable force-pushed the test-framework-extensions branch from 72953d0 to 69cead5 Compare September 16, 2020 16:09

randomvariable changed the title ~~test framework: Add kubetest, kubernetesversions, log and ginkgo extensions~~ ✨ test framework: Add kubetest, kubernetesversions, log and ginkgo extensions Sep 16, 2020

vincepri reviewed Sep 16, 2020

View reviewed changes

test/framework/kubernetesversions/zz_generated.bindata.go Show resolved Hide resolved

test/framework/kubernetesversions/zz_generated.bindata.go Show resolved Hide resolved

test/framework/kubernetesversions/zz_generated.bindata.go Show resolved Hide resolved

k8s-ci-robot assigned sedefsavas Sep 16, 2020

sedefsavas reviewed Sep 16, 2020

View reviewed changes

randomvariable force-pushed the test-framework-extensions branch 7 times, most recently from 1eb6652 to 585c2d6 Compare September 17, 2020 11:02

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 17, 2020

k8s-ci-robot assigned benmoss Sep 23, 2020

fabriziopandini reviewed Sep 24, 2020

View reviewed changes

randomvariable force-pushed the test-framework-extensions branch from 585c2d6 to cab26d5 Compare September 24, 2020 09:55

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 24, 2020

randomvariable force-pushed the test-framework-extensions branch from cab26d5 to 995578c Compare September 24, 2020 10:00

k8s-ci-robot added this to the v0.3.10 milestone Sep 24, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Sep 25, 2020

randomvariable mentioned this pull request Sep 25, 2020

Add e2e testing of cloud provider via Cluster API Clusterctl framework or Cluster API kubetest2 deployer kubernetes/cloud-provider-aws#130

Open

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Sep 25, 2020

k8s-ci-robot merged commit 59faa7b into kubernetes-sigs:master Sep 25, 2020

randomvariable deleted the test-framework-extensions branch September 28, 2020 10:55

randomvariable changed the title ~~✨ test framework: Add kubetest, kubernetesversions, log and ginkgo extensions~~ ✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions Sep 28, 2020

fabriziopandini mentioned this pull request Oct 30, 2020

Expand test framework to include upstream k8s testing #2826

Closed

fabriziopandini mentioned this pull request Nov 10, 2020

Add tests for CAPI e2e for main branch of Kubernetes #3569

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions #3652

✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions #3652

randomvariable commented Sep 16, 2020 •

edited

Loading

randomvariable commented Sep 16, 2020

vincepri left a comment

randomvariable commented Sep 17, 2020

randomvariable commented Sep 17, 2020

sedefsavas commented Sep 17, 2020

benmoss commented Sep 23, 2020

benmoss commented Sep 23, 2020

fabriziopandini left a comment

fabriziopandini Sep 24, 2020

randomvariable Sep 24, 2020

fabriziopandini Sep 24, 2020

randomvariable Sep 24, 2020

randomvariable Sep 24, 2020

CecileRobertMichon Oct 1, 2020

randomvariable Oct 1, 2020

randomvariable Oct 1, 2020

randomvariable commented Sep 24, 2020

k8s-ci-robot commented Sep 24, 2020 •

edited

Loading

vincepri commented Sep 24, 2020

randomvariable commented Sep 25, 2020

sedefsavas commented Sep 25, 2020

vincepri commented Sep 25, 2020

k8s-ci-robot commented Sep 25, 2020

vincepri commented Sep 25, 2020

k8s-ci-robot commented Sep 25, 2020

✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions #3652

✨ test framework: Add kubetest, kubernetesversions, and ginkgo extensions #3652

Conversation

randomvariable commented Sep 16, 2020 • edited Loading

randomvariable commented Sep 16, 2020

vincepri left a comment

Choose a reason for hiding this comment

randomvariable commented Sep 17, 2020

randomvariable commented Sep 17, 2020

sedefsavas commented Sep 17, 2020

benmoss commented Sep 23, 2020

benmoss commented Sep 23, 2020

fabriziopandini left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

randomvariable commented Sep 24, 2020

k8s-ci-robot commented Sep 24, 2020 • edited Loading

vincepri commented Sep 24, 2020

randomvariable commented Sep 25, 2020

sedefsavas commented Sep 25, 2020

vincepri commented Sep 25, 2020

k8s-ci-robot commented Sep 25, 2020

vincepri commented Sep 25, 2020

k8s-ci-robot commented Sep 25, 2020

randomvariable commented Sep 16, 2020 •

edited

Loading

k8s-ci-robot commented Sep 24, 2020 •

edited

Loading