Download all binaries instead of building them locally for compat version tests #34930

michaelasp · 2025-06-05T21:08:56Z

This pr downloads all required docker and test binaries and uses those to run the tests rather than building them locally. Running these tests locally showed a major performance improvement from this.

The two major changes are downloading all the control plane docker images rather than building them from source and also downloading the test binaries instead of building them to run conformance tests.

…on version tests

…ulation version tests

k8s-ci-robot · 2025-06-05T21:09:05Z

Hi @michaelasp. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

k8s-ci-robot · 2025-06-05T21:09:07Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: michaelasp
Once this PR has been reviewed and has the lgtm label, please assign aaron-prindle for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

experiment/compatibility-versions/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

michaelasp · 2025-06-05T21:09:42Z

/cc @BenTheElder

BenTheElder · 2025-06-10T20:57:56Z

experiment/compatibility-versions/kind-upgrade.sh

-build_docker(){
-  build/run.sh make all WHAT="cmd/kubectl cmd/kubelet" 1> /dev/null
-  make quick-release-images 1> /dev/null
+download_docker(){


this name is misleading (the old one is too, it's a carry over from when we had a bazel build mode, but this is even more misleading)

maybe "download_images"

BenTheElder · 2025-06-10T20:59:44Z

experiment/compatibility-versions/common.sh

+}
+
+download_current_version_bins() {
+  wget https://dl.k8s.io/ci/$(curl -Ls https://dl.k8s.io/ci/latest.txt)/kubernetes-test-$(go env GOOS)-$(go env GOARCH).tar.gz


maybe something like this:

Suggested change

wget https://dl.k8s.io/ci/$(curl -Ls https://dl.k8s.io/ci/latest.txt)/kubernetes-test-$(go env GOOS)-$(go env GOARCH).tar.gz

wget 'https://dl.k8s.io/ci/'"${1}"/'kubernetes-test-'"$(go env GOOS)-$(go env GOARCH)"'.tar.gz'

and pass in the version (see below)

Makes sense, this can make for some nasty edge cases. I made it so only the create_cluster step obtains the latest_version. I think that's the best place to put it in common.sh since we need to create a cluster in every test and we have access to artifacts at that point.

BenTheElder · 2025-06-10T21:00:44Z

experiment/compatibility-versions/common.sh

@@ -235,6 +235,30 @@ build_prev_version_bins() {
  echo "Finished building e2e.test binary from ${PREV_RELEASE_BRANCH}."
 }

+download_prev_version_bins() {
+  local version=$1
+  wget https://dl.k8s.io/release/$(curl -Ls https://dl.k8s.io/release/stable-${version}.txt)/kubernetes-test-$(go env GOOS)-$(go env GOARCH).tar.gz


TODO: this script as-written will only work on a linux host, but it should work on mac too.

For that you need to split binaries being downloaded for kind (GOOS should always be linux) versus for the host (like the test binaries)

Added a todo :)

BenTheElder · 2025-06-10T21:01:49Z

experiment/compatibility-versions/kind-upgrade.sh

@@ -32,8 +32,9 @@ CONTROL_PLANE_COMPONENTS="kube-apiserver kube-controller-manager kube-scheduler"
 KUBE_ROOT="."
 # KUBE_ROOT="$(go env GOPATH)/src/k8s.io/kubernetes"
 source "${KUBE_ROOT}/hack/lib/version.sh"
-kube::version::get_version_vars
-DOCKER_TAG=${KUBE_GIT_VERSION/+/_}
+LATEST_IMAGE=$(curl -Ls https://dl.k8s.io/ci/latest.txt)


we should make sure to use this value when downloading the current version, otherwise we're going to wind up with a race where we record one version and test another.

Suggested change

LATEST_IMAGE=$(curl -Ls https://dl.k8s.io/ci/latest.txt)

LATEST_VERSION=$(curl -Ls https://dl.k8s.io/ci/latest.txt)

Ack, made it so only the common script modifies it in the create cluster step.

BenTheElder · 2025-06-10T21:02:16Z

Thanks, sorry for the delay (PRR crunch...), some comments above ...

BenTheElder · 2025-06-10T21:04:26Z

experiment/compatibility-versions/kind-upgrade.sh

-  make quick-release-images 1> /dev/null
+download_docker(){
+  curl -L 'https://dl.k8s.io/ci/'${LATEST_IMAGE}'/kubernetes-server-linux-amd64.tar.gz' > kubernetes-server-linux-amd64.tar.gz
+  tar -xvf kubernetes-server-linux-amd64.tar.gz


You probably want to stage files in a tempdir instead of in the current working directory, it can make the repo dirty (also, they might not get cleaned up ...)

This server tarball can also be used to build a node image, so we don't need to checkout and build k/k at all.

kind build node-image --type file kubernetes-server-linux-amd64.tar.gz
https://kind.sigs.k8s.io/docs/user/quick-start/#building-images

Done! I don't think we build k/k at all here, the create cluster step uses a precompiled image. I did however remove the cloning of the k8s repo, this is no longer necessary since we pull the precompiled binary.

michaelasp · 2025-06-17T18:10:26Z

/cc @Jefftree

michaelasp added 3 commits June 5, 2025 21:05

feat: Download docker binaries instead of building locally in emulati…

61c6e53

…on version tests

feat: Add ability to download instead of make e2e test binaries in em…

7cbaf78

…ulation version tests

feat: Download e2e binaries in all emulation version tests

eafd5cb

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label Jun 5, 2025

k8s-ci-robot requested review from aaron-prindle and aojea June 5, 2025 21:09

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jun 5, 2025

k8s-ci-robot added the sig/testing Categorizes an issue or PR as relevant to SIG Testing. label Jun 5, 2025

k8s-ci-robot requested a review from BenTheElder June 5, 2025 21:09

michaelasp mentioned this pull request Jun 5, 2025

feat: Add test to prow for skip version testing #34912

Merged

BenTheElder reviewed Jun 10, 2025

View reviewed changes

Address comments

58e8012

michaelasp force-pushed the nobuild branch from 3583fb5 to 58e8012 Compare June 10, 2025 23:38

michaelasp requested a review from BenTheElder June 11, 2025 20:12

k8s-ci-robot requested a review from Jefftree June 17, 2025 18:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Download all binaries instead of building them locally for compat version tests #34930

Download all binaries instead of building them locally for compat version tests #34930

michaelasp commented Jun 5, 2025

Uh oh!

k8s-ci-robot commented Jun 5, 2025

Uh oh!

k8s-ci-robot commented Jun 5, 2025

Uh oh!

michaelasp commented Jun 5, 2025

Uh oh!

BenTheElder Jun 10, 2025

Uh oh!

michaelasp Jun 10, 2025

Uh oh!

BenTheElder Jun 10, 2025

Uh oh!

michaelasp Jun 10, 2025

Uh oh!

BenTheElder Jun 10, 2025

Uh oh!

michaelasp Jun 10, 2025

Uh oh!

BenTheElder Jun 10, 2025

Uh oh!

michaelasp Jun 10, 2025

Uh oh!

BenTheElder commented Jun 10, 2025

Uh oh!

BenTheElder Jun 10, 2025

Uh oh!

michaelasp Jun 10, 2025 •

edited

Loading

Uh oh!

michaelasp commented Jun 17, 2025

Uh oh!

Uh oh!

	wget https://dl.k8s.io/ci/$(curl -Ls https://dl.k8s.io/ci/latest.txt)/kubernetes-test-$(go env GOOS)-$(go env GOARCH).tar.gz
	wget 'https://dl.k8s.io/ci/'"${1}"/'kubernetes-test-'"$(go env GOOS)-$(go env GOARCH)"'.tar.gz'

	LATEST_IMAGE=$(curl -Ls https://dl.k8s.io/ci/latest.txt)
	LATEST_VERSION=$(curl -Ls https://dl.k8s.io/ci/latest.txt)

Download all binaries instead of building them locally for compat version tests #34930

Are you sure you want to change the base?

Download all binaries instead of building them locally for compat version tests #34930

Conversation

michaelasp commented Jun 5, 2025

Uh oh!

k8s-ci-robot commented Jun 5, 2025

Uh oh!

k8s-ci-robot commented Jun 5, 2025

Uh oh!

michaelasp commented Jun 5, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

BenTheElder commented Jun 10, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelasp Jun 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

michaelasp commented Jun 17, 2025

Uh oh!

Uh oh!

michaelasp Jun 10, 2025 •

edited

Loading