🌱 Add scale testing for upgrades #9077

killianmuldoon · 2023-07-25T18:43:13Z

Add upgrades to the Cluster API scale tests.

Part of #8814

test/e2e/scale_test.go

killianmuldoon · 2023-07-25T18:45:31Z

test/framework/cluster_topology_helpers.go

-		KubernetesVersion: input.KubernetesUpgradeVersion,
-	}, input.WaitForKubeProxyUpgrade...)
+
+	if !input.SkipKubeProxyCheck {


Added this knob on the testing input to allow skipping this check. This requires routing from the host to the port on the in-memory provider pod which represents the API server of the workload cluster. I didn't figure out a way to enable this, but in this iteration upgrade checks for etcd, kube-proxy and coreDNS do not work.

killianmuldoon · 2023-07-25T18:47:08Z

test/framework/machine_helpers.go

@@ -127,7 +127,13 @@ func GetControlPlaneMachinesByCluster(ctx context.Context, input GetControlPlane
 	Expect(input.ClusterName).ToNot(BeEmpty(), "Invalid argument. input.ClusterName can't be empty when calling GetControlPlaneMachinesByCluster")
 	Expect(input.Namespace).ToNot(BeEmpty(), "Invalid argument. input.Namespace can't be empty when calling GetControlPlaneMachinesByCluster")

-	options := append(byClusterOptions(input.ClusterName, input.Namespace), controlPlaneMachineOptions()...)


AFAICT this never worked with multiple clusters as one label selector overrode the other - this ended up getting all machineDeployments in the namespace for me. Replaced it with a simpler inline definition of the label selector.

Fun! nice catch!

killianmuldoon · 2023-07-25T18:47:30Z

test/infrastructure/inmemory/internal/server/api/handler.go

 	if err != nil {
 		_ = resp.WriteErrorString(http.StatusInternalServerError, err.Error())
 		return
 	}
-	if selector != nil {
-		listOpts = append(listOpts, client.MatchingFieldsSelector{Selector: selector})
+	if fieldSelector != nil {


This is incidental, but I fixed it up in the course of the PR and decided to keep it. Can split it out if needed.

killianmuldoon · 2023-11-07T17:02:53Z

The basic machinery is in place here - I think we should move forward with trying to merge it as-is and take the problems, as follow-ups.

killianmuldoon

/test

k8s-ci-robot · 2023-11-07T17:07:38Z

@killianmuldoon: The /test command needs one or more targets.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-main
/test pull-cluster-api-e2e-full-dualstack-and-ipv6-main
/test pull-cluster-api-e2e-full-main
/test pull-cluster-api-e2e-main
/test pull-cluster-api-e2e-mink8s-main
/test pull-cluster-api-e2e-workload-upgrade-1-28-latest-main
/test pull-cluster-api-test-main
/test pull-cluster-api-test-mink8s-main
/test pull-cluster-api-verify-main

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-main
/test pull-cluster-api-e2e-scale-main-experimental

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-apidiff-main
pull-cluster-api-build-main
pull-cluster-api-e2e-main
pull-cluster-api-test-main
pull-cluster-api-verify-main

In response to this:

/test

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

killianmuldoon · 2023-11-07T17:07:48Z

/test pull-cluster-api-e2e-full-main

sbueringer · 2023-12-13T08:16:51Z

@killianmuldoon Can you please rebase?

I'll try to find time to review right after

killianmuldoon · 2024-01-22T18:15:42Z

@sbueringer - sorry it took so long to get back to this. I'll spend some cycles on it if you get time to review though.

sbueringer

Thx for following-up to!

test/e2e/scale.go

sbueringer · 2024-01-25T15:12:39Z

test/framework/machine_helpers.go

@@ -127,7 +127,13 @@ func GetControlPlaneMachinesByCluster(ctx context.Context, input GetControlPlane
 	Expect(input.ClusterName).ToNot(BeEmpty(), "Invalid argument. input.ClusterName can't be empty when calling GetControlPlaneMachinesByCluster")
 	Expect(input.Namespace).ToNot(BeEmpty(), "Invalid argument. input.Namespace can't be empty when calling GetControlPlaneMachinesByCluster")

-	options := append(byClusterOptions(input.ClusterName, input.Namespace), controlPlaneMachineOptions()...)


Fun! nice catch!

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

sbueringer · 2024-01-29T10:08:09Z

Thank you very much!

Really appreciate that you followed-up on this :)

/test pull-cluster-api-e2e-main
(to actually run the test ;))

/lgtm
/approve

k8s-ci-robot · 2024-01-29T10:08:15Z

LGTM label has been added.

Git tree hash: d30effd733b14d1e37ab9c66f4d13866cda8f2b7

k8s-ci-robot · 2024-01-29T10:08:18Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sbueringer

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [sbueringer]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

typeid · 2024-04-08T14:40:41Z

/area testing

k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/L Denotes a PR that changes 100-499 lines, ignoring generated files. labels Jul 25, 2023

k8s-ci-robot requested review from chrischdi and jackfrancis July 25, 2023 18:43

killianmuldoon force-pushed the pr-scale-test-upgrade branch from 846b51a to d4f69ad Compare July 25, 2023 18:46

killianmuldoon commented Jul 25, 2023

View reviewed changes

killianmuldoon force-pushed the pr-scale-test-upgrade branch from d4f69ad to 96b9c5a Compare July 25, 2023 18:53

k8s-ci-robot added cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. and removed cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Jul 26, 2023

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Sep 29, 2023

killianmuldoon force-pushed the pr-scale-test-upgrade branch from 96b9c5a to 47676c9 Compare November 7, 2023 17:00

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. and removed cncf-cla: no Indicates the PR's author has not signed the CNCF CLA. needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. labels Nov 7, 2023

killianmuldoon changed the title ~~[WIP] 🌱 Add Upgrade flow to scale test~~ 🌱 Add scale testing for upgrades Nov 7, 2023

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Nov 7, 2023

killianmuldoon commented Nov 7, 2023

View reviewed changes

killianmuldoon mentioned this pull request Nov 9, 2023

Replace kubectl apply in e2e test framework #9696

Closed

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 12, 2023

killianmuldoon force-pushed the pr-scale-test-upgrade branch from 47676c9 to b1598e8 Compare January 22, 2024 18:15

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 23, 2024

sbueringer reviewed Jan 25, 2024

View reviewed changes

k8s-ci-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 25, 2024

Add Upgrade flow to scale test

71bc6fe

Signed-off-by: killianmuldoon <kmuldoon@vmware.com>

killianmuldoon force-pushed the pr-scale-test-upgrade branch from b1598e8 to 71bc6fe Compare January 28, 2024 12:15

k8s-ci-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Jan 28, 2024

k8s-ci-robot assigned sbueringer Jan 29, 2024

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jan 29, 2024

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 29, 2024

k8s-ci-robot merged commit 6bbcfda into kubernetes-sigs:main Jan 29, 2024
21 checks passed

k8s-ci-robot added this to the v1.7 milestone Jan 29, 2024

k8s-ci-robot added the area/testing Issues or PRs related to testing label Apr 8, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🌱 Add scale testing for upgrades #9077

🌱 Add scale testing for upgrades #9077

killianmuldoon commented Jul 25, 2023

killianmuldoon Jul 25, 2023 •

edited

Loading

killianmuldoon Jul 25, 2023

sbueringer Jan 25, 2024

killianmuldoon Jul 25, 2023

killianmuldoon commented Nov 7, 2023

killianmuldoon left a comment

k8s-ci-robot commented Nov 7, 2023

killianmuldoon commented Nov 7, 2023

sbueringer commented Dec 13, 2023

killianmuldoon commented Jan 22, 2024

sbueringer left a comment

sbueringer Jan 25, 2024

sbueringer commented Jan 29, 2024

k8s-ci-robot commented Jan 29, 2024

k8s-ci-robot commented Jan 29, 2024

typeid commented Apr 8, 2024

🌱 Add scale testing for upgrades #9077

🌱 Add scale testing for upgrades #9077

Conversation

killianmuldoon commented Jul 25, 2023

killianmuldoon Jul 25, 2023 • edited Loading

Choose a reason for hiding this comment

killianmuldoon Jul 25, 2023

Choose a reason for hiding this comment

sbueringer Jan 25, 2024

Choose a reason for hiding this comment

killianmuldoon Jul 25, 2023

Choose a reason for hiding this comment

killianmuldoon commented Nov 7, 2023

killianmuldoon left a comment

Choose a reason for hiding this comment

k8s-ci-robot commented Nov 7, 2023

killianmuldoon commented Nov 7, 2023

sbueringer commented Dec 13, 2023

killianmuldoon commented Jan 22, 2024

sbueringer left a comment

Choose a reason for hiding this comment

sbueringer Jan 25, 2024

Choose a reason for hiding this comment

sbueringer commented Jan 29, 2024

k8s-ci-robot commented Jan 29, 2024

k8s-ci-robot commented Jan 29, 2024

typeid commented Apr 8, 2024

killianmuldoon Jul 25, 2023 •

edited

Loading