✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0 #5013

sbueringer · 2021-07-23T20:04:57Z

What this PR does / why we need it:
See linked issue

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Partially implements #4966

controlplane/kubeadm/controllers/controller.go

sbueringer · 2021-07-26T15:09:37Z

@vincepri @fabriziopandini PTAL :)

sbueringer · 2021-07-26T15:22:41Z

/test pull-cluster-api-test-release-0-3
(flake afaik)

sbueringer · 2021-07-26T15:30:00Z

/retest
(I assume it's flaky as I tested the PR locally)

sbueringer · 2021-07-26T15:43:30Z

Okay nope, moving it up one layer broke the tests. I'll have to investigate.

sbueringer · 2021-07-26T16:35:18Z

My local env on the old branch with kubebuilder does not really work (even with make test, I assume the problem is that Mac Security is blocking a lot of stuff). So I'll probably need a few tries to get this fixed...

sbueringer · 2021-07-26T16:58:42Z

/test pull-cluster-api-e2e-full-release-0-3

sbueringer · 2021-07-26T17:04:52Z

@fabriziopandini Should be ready for review now. I moved the func in the WorkloadCluster interface as it's just a lot easier to test that way. I'm not sure what I would have to refactor to get the regular reconcile unit tests working again when I add it in one of the reconcile funcs.

As I'm not that familiar with the release-0.3 e2e tests. Is the test coverage good enough or do you think I should test something manually?

/test pull-cluster-api-e2e-full-release-0-3

fabriziopandini

/lgtm

E2E could provide a signal that the change does not introduces regression, but it would be great if you could do a manual test with 1.20, then upgrade to 1.21 (everything should work), then upgrade to 1.22 (it should block)

controlplane/kubeadm/internal/workload_cluster.go

controlplane/kubeadm/internal/cluster.go

controlplane/kubeadm/internal/workload_cluster.go

sbueringer · 2021-07-28T16:33:52Z

/retest

looks like flakes to me

@vincepri Thx for the review :). Findings should be fixed, ptal.

controlplane/kubeadm/internal/cluster.go

sbueringer · 2021-07-28T18:34:49Z

updated the list of uncached types (although I'm not entirely sure it includes all necessary types). I brute-force the flakes successful after reviews :) (unit tests were successful locally)

vincepri · 2021-07-28T23:36:59Z

/retest

controlplane/kubeadm/controllers/upgrade.go

vincepri · 2021-07-28T23:39:10Z

controlplane/kubeadm/internal/cluster.go

+		return nil, &RemoteClusterConnectionError{Name: clusterKey.String(), Err: err}
+	}
+
+	newClient := util.DelegatingClientFuncWithUncached(


@sbueringer Sorry for the confusion, could we actually try to just use the plain client.New without the delegating client and retrieve with PartialObjectMetadata directly? It should work considering that it has support for it https://github.com/kubernetes-sigs/controller-runtime/blob/release-0.5/pkg/client/client.go#L111

Oh, yeah will try. At least I'm learning a lot :)

Works, tested it locally with all relevant code branches of IsKubernetesVersionSupported and it uses the metaclient as expected.

sbueringer · 2021-07-29T12:30:55Z

/test pull-cluster-api-e2e-full-release-0-3

sbueringer · 2021-07-29T13:28:05Z

/test pull-cluster-api-test-release-0.3

@fabriziopandini Tested it locally and it worked. There were a few interesting pitfalls (cgroup-driver and local dev image). Thus, for reference:

Prepare KCP image

export REGISTRY=gcr.io/k8s-staging-cluster-api
export TAG=dev
export ARCH=amd64
export PULL_POLICY=IfNotPresent
make docker-build

Setup Management cluster

kind create cluster --config /tmp/kind-cluster-with-extramounts.yaml
clusterctl-v0.3.22 init --infrastructure docker
# use KCP dev image
kind load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kind get kubeconfig --name=$(kind get clusters | grep kind) | k8s-ctx-import; kctx kind-$(kind get clusters | grep kind)
kubectl patch deployment -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager  --type json   -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/image", value: "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"}]'

Install v1.20.7 Workload Cluster

clusterctl-v0.3.22 config cluster capi-quickstart --kubernetes-version v1.20.7 --control-plane-machine-count=1 --worker-machine-count=1 --flavor=development > capi-quickstart.yaml
# Edit capi-quickstart.yaml to add `cgroup-driver: cgroupfs` kubeletExtraArgs
kubectl -n default apply -f ./capi-quickstart.yaml
kubectl -n default get secret capi-quickstart-kubeconfig  -o json | jq '.data.value' -r | base64 -d > /tmp/kubeconfig
sed -i -e "s/certificate-authority-data:.*/insecure-skip-tls-verify: true/g" /tmp/kubeconfig
sed -i -e "s/server:.*/server: https:\/\/$(docker port capi-quickstart-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g"  /tmp/kubeconfig
kubectl --kubeconfig=/tmp/kubeconfig apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml

Pivot workload cluster to self-hosted management cluster

clusterctl-v0.3.22 init --infrastructure docker --kubeconfig=/tmp/kubeconfig
# use KCP dev image
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kubectl --kubeconfig=/tmp/kubeconfig patch deployment -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager  --type json   -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/image", value: "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"}]'
clusterctl-v0.3.22 move --to-kubeconfig=/tmp/kubeconfig

Update to v1.21.2

kubectl --kubeconfig=/tmp/kubeconfig patch kcp capi-quickstart-control-plane --type json -p='[{"op": "replace", "path": "/spec/version", value: "v1.21.2"}]'
# load KCP dev image on the new node
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kubectl --kubeconfig=/tmp/kubeconfig patch md capi-quickstart-md-0 --type json -p='[{"op": "replace", "path": "/spec/template/spec/version", value: "v1.21.2"}]'
# load KCP dev image on the new node
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"

Update to v1.22.0

kubectl --kubeconfig=/tmp/kubeconfig patch kcp capi-quickstart-control-plane --type json -p='[{"op": "replace", "path": "/spec/version", value: "v1.22.0"}]'

Update to 1.21 worked and the last step resulted in the following log in KCP:

I0729 13:25:34.603463       1 upgrade.go:59]  "msg"="Kubernetes version \"v1.22.0\" is not supported for management clusters" "cluster-name"="capi-quickstart" "name"="capi-quickstart-control-plane" "namespace"="default"

k8s-ci-robot · 2021-07-29T13:28:07Z

@sbueringer: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-release-0-3
/test pull-cluster-api-make-release-0-3
/test pull-cluster-api-verify-release-0-3
/test pull-cluster-api-test-release-0-3
/test pull-cluster-api-e2e-release-0-3

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-release-0-3
/test pull-cluster-api-e2e-full-release-0-3

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-build-release-0-3
pull-cluster-api-make-release-0-3
pull-cluster-api-apidiff-release-0-3
pull-cluster-api-verify-release-0-3
pull-cluster-api-test-release-0-3
pull-cluster-api-e2e-release-0-3

In response to this:

/test pull-cluster-api-test-release-0.3

@fabriziopandini Tested it locally and it worked. There were a few interesting pitfalls (cgroup-driver and local dev image). Thus, for reference:

Prepare KCP image

export REGISTRY=gcr.io/k8s-staging-cluster-api
export TAG=dev
export ARCH=amd64
export PULL_POLICY=IfNotPresent
make docker-build

Setup Management cluster

kind create cluster --config /tmp/kind-cluster-with-extramounts.yaml
clusterctl-v0.3.22 init --infrastructure docker
# use KCP dev image
kind load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kind get kubeconfig --name=$(kind get clusters | grep kind) | k8s-ctx-import; kctx kind-$(kind get clusters | grep kind)
kubectl patch deployment -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager  --type json   -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/image", value: "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"}]'

Install v1.20.7 Workload Cluster

clusterctl-v0.3.22 config cluster capi-quickstart --kubernetes-version v1.20.7 --control-plane-machine-count=1 --worker-machine-count=1 --flavor=development > capi-quickstart.yaml
# Edit capi-quickstart.yaml to add `cgroup-driver: cgroupfs` kubeletExtraArgs
kubectl -n default apply -f ./capi-quickstart.yaml
kubectl -n default get secret capi-quickstart-kubeconfig  -o json | jq '.data.value' -r | base64 -d > /tmp/kubeconfig
sed -i -e "s/certificate-authority-data:.*/insecure-skip-tls-verify: true/g" /tmp/kubeconfig
sed -i -e "s/server:.*/server: https:\/\/$(docker port capi-quickstart-lb 6443/tcp | sed "s/0.0.0.0/127.0.0.1/")/g"  /tmp/kubeconfig
kubectl --kubeconfig=/tmp/kubeconfig apply -f https://docs.projectcalico.org/v3.15/manifests/calico.yaml

Pivot workload cluster to self-hosted management cluster

clusterctl-v0.3.22 init --infrastructure docker --kubeconfig=/tmp/kubeconfig
# use KCP dev image
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kubectl --kubeconfig=/tmp/kubeconfig patch deployment -n capi-kubeadm-control-plane-system capi-kubeadm-control-plane-controller-manager  --type json   -p='[{"op": "replace", "path": "/spec/template/spec/containers/1/image", value: "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"}]'
clusterctl-v0.3.22 move --to-kubeconfig=/tmp/kubeconfig

Update to v1.21.2

kubectl --kubeconfig=/tmp/kubeconfig patch kcp capi-quickstart-control-plane --type json -p='[{"op": "replace", "path": "/spec/version", value: "v1.21.2"}]'
# load KCP dev image on the new node
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"
kubectl --kubeconfig=/tmp/kubeconfig patch md capi-quickstart-md-0 --type json -p='[{"op": "replace", "path": "/spec/template/spec/version", value: "v1.21.2"}]'
# load KCP dev image on the new node
kind --name capi-quickstart load docker-image "gcr.io/k8s-staging-cluster-api/kubeadm-control-plane-controller-amd64:dev"

Update to v1.22.0

kubectl --kubeconfig=/tmp/kubeconfig patch kcp capi-quickstart-control-plane --type json -p='[{"op": "replace", "path": "/spec/version", value: "v1.22.0"}]'

The last step resulted in the following log in KCP:

I0729 13:25:34.603463       1 upgrade.go:59]  "msg"="Kubernetes version \"v1.22.0\" is not supported for management clusters" "cluster-name"="capi-quickstart" "name"="capi-quickstart-control-plane" "namespace"="default"

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2021-07-29T13:30:01Z

/test pull-cluster-api-test-release-0-3

sbueringer · 2021-07-29T13:37:17Z

/test @k8s-ci-robot
pull-cluster-api-test-release-0-3

k8s-ci-robot · 2021-07-29T13:37:19Z

@sbueringer: The specified target(s) for /test were not found.
The following commands are available to trigger required jobs:

/test pull-cluster-api-build-release-0-3
/test pull-cluster-api-make-release-0-3
/test pull-cluster-api-verify-release-0-3
/test pull-cluster-api-test-release-0-3
/test pull-cluster-api-e2e-release-0-3

The following commands are available to trigger optional jobs:

/test pull-cluster-api-apidiff-release-0-3
/test pull-cluster-api-e2e-full-release-0-3

Use /test all to run the following jobs that were automatically triggered:

pull-cluster-api-build-release-0-3
pull-cluster-api-make-release-0-3
pull-cluster-api-apidiff-release-0-3
pull-cluster-api-verify-release-0-3
pull-cluster-api-test-release-0-3
pull-cluster-api-e2e-release-0-3

In response to this:

/test @k8s-ci-robot
pull-cluster-api-test-release-0-3

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

sbueringer · 2021-07-29T13:44:11Z

/test pull-cluster-api-test-release-0-3

vincepri · 2021-07-29T14:24:43Z

@sbueringer Let's squash :)

Signed-off-by: Stefan Büringer buringerst@vmware.com

sbueringer · 2021-07-29T14:52:11Z

@sbueringer Let's squash :)

Done :)

vincepri

/approve
/lgtm

k8s-ci-robot · 2021-07-29T17:14:55Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: vincepri

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [vincepri]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added this to the v0.3 milestone Jul 23, 2021

k8s-ci-robot added cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 23, 2021

k8s-ci-robot requested review from benmoss and justinsb July 23, 2021 20:05

sbueringer commented Jul 23, 2021

View reviewed changes

controlplane/kubeadm/controllers/controller.go Outdated Show resolved Hide resolved

vincepri reviewed Jul 23, 2021

View reviewed changes

controlplane/kubeadm/controllers/controller.go Outdated Show resolved Hide resolved

sbueringer commented Jul 23, 2021

View reviewed changes

controlplane/kubeadm/controllers/controller.go Outdated Show resolved Hide resolved

sbueringer force-pushed the pr-kcp-block-mgmt-cluster-updates branch from 3d081f6 to 14ec45d Compare July 26, 2021 15:04

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 26, 2021

sbueringer force-pushed the pr-kcp-block-mgmt-cluster-updates branch from 14ec45d to 0ea76df Compare July 26, 2021 15:09

sbueringer changed the title ~~✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0~~ [WIP] ✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0 Jul 26, 2021

k8s-ci-robot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 26, 2021

sbueringer changed the title ~~[WIP] ✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0~~ ✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0 Jul 26, 2021

k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Jul 26, 2021

fabriziopandini reviewed Jul 28, 2021

View reviewed changes

k8s-ci-robot assigned fabriziopandini Jul 28, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 28, 2021

vincepri reviewed Jul 28, 2021

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Outdated Show resolved Hide resolved

vincepri reviewed Jul 28, 2021

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Outdated Show resolved Hide resolved

vincepri reviewed Jul 28, 2021

View reviewed changes

controlplane/kubeadm/internal/cluster.go Outdated Show resolved Hide resolved

k8s-ci-robot removed the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 28, 2021

sbueringer commented Jul 28, 2021

View reviewed changes

controlplane/kubeadm/internal/workload_cluster.go Show resolved Hide resolved

sbueringer commented Jul 28, 2021

View reviewed changes

controlplane/kubeadm/internal/cluster.go Outdated Show resolved Hide resolved

vincepri reviewed Jul 28, 2021

View reviewed changes

KCP: block upgrade of management clusters to versions >= v1.22.0

8456ae8

Signed-off-by: Stefan Büringer buringerst@vmware.com

sbueringer force-pushed the pr-kcp-block-mgmt-cluster-updates branch from 95f8b2c to 8456ae8 Compare July 29, 2021 14:51

vincepri approved these changes Jul 29, 2021

View reviewed changes

k8s-ci-robot assigned vincepri Jul 29, 2021

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 29, 2021

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 29, 2021

k8s-ci-robot merged commit 1c68fe9 into kubernetes-sigs:release-0.3 Jul 29, 2021

sbueringer deleted the pr-kcp-block-mgmt-cluster-updates branch July 29, 2021 17:24

✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0 #5013

✨ v0.3.x: KCP: block mgmt cluster updates to v1.22.0 #5013

Uh oh!

Conversation

sbueringer commented Jul 23, 2021

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Jul 26, 2021

Uh oh!

sbueringer commented Jul 26, 2021

Uh oh!

sbueringer commented Jul 26, 2021

Uh oh!

sbueringer commented Jul 26, 2021

Uh oh!

sbueringer commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sbueringer commented Jul 26, 2021

Uh oh!

sbueringer commented Jul 26, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

fabriziopandini left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

sbueringer commented Jul 28, 2021

Uh oh!

Uh oh!

sbueringer commented Jul 28, 2021

Uh oh!

vincepri commented Jul 28, 2021

Uh oh!

Uh oh!

vincepri Jul 28, 2021

Choose a reason for hiding this comment

Uh oh!

sbueringer Jul 29, 2021

Choose a reason for hiding this comment

Uh oh!

sbueringer Jul 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sbueringer commented Jul 29, 2021

Uh oh!

sbueringer commented Jul 29, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Prepare KCP image

Setup Management cluster

Install v1.20.7 Workload Cluster

Pivot workload cluster to self-hosted management cluster

Update to v1.21.2

Update to v1.22.0

Uh oh!

k8s-ci-robot commented Jul 29, 2021

Prepare KCP image

Setup Management cluster

Install v1.20.7 Workload Cluster

Pivot workload cluster to self-hosted management cluster

Update to v1.21.2

Update to v1.22.0

Uh oh!

sbueringer commented Jul 29, 2021

Uh oh!

sbueringer commented Jul 29, 2021

Uh oh!

k8s-ci-robot commented Jul 29, 2021

Uh oh!

sbueringer commented Jul 29, 2021

Uh oh!

vincepri commented Jul 29, 2021

Uh oh!

sbueringer commented Jul 29, 2021

sbueringer commented Jul 26, 2021 •

edited

Loading

sbueringer commented Jul 26, 2021 •

edited

Loading

sbueringer Jul 29, 2021 •

edited

Loading

sbueringer commented Jul 29, 2021 •

edited

Loading