Bump e2e test k8s versions

[mboersma]

What type of PR is this?

/kind cleanup

What this PR does / why we need it:

Updates Kubernetes versions under test to newer versions.

I bumped Flatcar too, hopefully that works! Nope, reverted.

Which issue(s) this PR fixes:

See #4434 for the previous time these were bumped.

Special notes for your reviewer:

• cherry-pick candidate

TODOs:

• squashed commits
• includes documentation
• adds unit tests

Release note:

NONE

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 62.03%. Comparing base (9d3fa78) to head (8004878).
Report is 2 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4865      +/-   ##
==========================================
- Coverage   62.04%   62.03%   -0.02%     
==========================================
  Files         201      201              
  Lines       16878    16878              
==========================================
- Hits        10472    10470       -2     
- Misses       5623     5625       +2     
  Partials      783      783              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[nojnhuh]

@mboersma In case you haven't tracked it down yet, I think the test is failing because the KubeadmControlPlane resource specifies feature-gates: WindowsHostProcessContainers=true for the API server and it looks like that feature flag is no longer valid in Kubernetes v1.28 because it went GA: https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/

[nojnhuh]

Kubebuilder is migrating their domain and causing the link checker to fail in the meantime:
https://kubernetes.slack.com/archives/CAR30FCJZ/p1716281938319289

[mboersma]

/test pull-cluster-api-provider-azure-e2e-optional
(for Flatcar)

[nojnhuh]

Thank you for also updating docs!

/lgtm
/approve
/hold for optional tests

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 55d52c386b98997a2216f93f6313b09494c9c8fa

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: nojnhuh

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [nojnhuh]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[mboersma]

Newer Flatcar image was located, so something else must have gone wrong:

  May 21 21:26:55.087: INFO: Resolved FLATCAR_KUBERNETES_VERSION (set to stable-1.27) to v1.27.8
  May 21 21:26:55.087: INFO: Finding Flatcar versions in community gallery "flatcar4capi-742ef0cb-dcaa-4ecb-9cb0-bfd2e43dccc0" in location "westus2" for image "flatcar-stable-amd64-capi-v1.27.8"
  May 21 21:26:55.516: INFO: Resolved FLATCAR_VERSION (set to latest) to 3602.2.2

I think I'll just revert the Flatcar bump for now.

[nojnhuh]

/test pull-cluster-api-provider-azure-e2e-optional
just for the sanity check on the flatcar test

/lgtm
keeping the hold for the optional test

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: b67e9b974ee4e42f3bb00ebf4e1be9c63ae96a49

[mboersma]

After reverting Flatcar to a known working version of Kubernetes (1.26 😞), the Workload ID optional test is still failing. Not sure why bumping the k8s version would affect this, but the kubelet log shows things aren't happy.

[nojnhuh]

Looks like yet another newly-GA feature gate:

cluster-api-provider-azure/templates/test/ci/cluster-template-prow-workload-identity.yaml

Line 60 in e0af1ae

feature-gates: MixedProtocolLBService=true

https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates-removed/

[mboersma]

You rock @nojnhuh, I was just starting to dig deeper but that looks like the culprit. MixedProtocolLBService was removed in 1.28, so I'll remove it as well from this PR.

For my edification, where did you find this in the logs? Was I just not looking closely enough?

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: 5865fc54a9923f2c601b3e19f17b408328fd107e

[nojnhuh]

/lgtm

Reminder there's still a hold on this for the optional tests which have passed before, so feel free to remove that whenever. I'm not too worried about the rebase affecting the optional tests.

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: c9e70f49ecf0486c567add819eef343def5e0a3c

[mboersma]

I'll run the -optional tests again before we merge it. Currently I'm more concerned with the -ci-artifacts failures that seem not to be flakes...

[mboersma]

/test pull-cluster-api-provider-azure-e2e-optional

[nojnhuh]

/lgtm

I expect the MEC/edge zone test in the optional job to still fail, but I don't think we should block this PR on that.

@mboersma Feel free to drop the hold when the optional job finishes and the commit history here looks however you want.

[k8s-ci-robot]

LGTM label has been added.

Git tree hash: c6ed8bb4d25069beec5bfd9beeb2277f8459a624

[mboersma]

the commit history

I thought I'd leave this as three separate commits, because I can imagine someone spelunking the codebase and wondering "When did CAPZ lose the WindowsHostProcessContainers feature flag?" for example.

[k8s-ci-robot]

@mboersma: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
pull-cluster-api-provider-azure-e2e-optional	8004878	link	false	/test pull-cluster-api-provider-azure-e2e-optional

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[mboersma]

/cherry-pick release-1.15
/cherry-pick release-1.14
/hold cancel

As expected, the MEC e2e test failed, but everything else is green.

[k8s-infra-cherrypick-robot]

@mboersma: once the present PR merges, I will cherry-pick it on top of release-1.15 in a new PR and assign it to you.

In response to this:

/cherry-pick release-1.15
/cherry-pick release-1.14
/hold cancel

As expected, the MEC e2e test failed, but everything else is green.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@mboersma: #4865 failed to apply on top of branch "release-1.15":

Applying: Bump e2e test k8s versions
Applying: Remove WindowsHostProcessContainers feature flag
Using index info to reconstruct a base tree...
A	test/e2e/data/infrastructure-azure/v1.14.5/cluster-template-prow.yaml
A	test/e2e/data/infrastructure-azure/v1.15.1/cluster-template-prow.yaml
Falling back to patching base and 3-way merge...
Applying: Remove MixedProtocolLBService feature flag
Using index info to reconstruct a base tree...
M	templates/test/ci/prow-edgezone/kustomization.yaml
M	templates/test/ci/prow-workload-identity/kustomization.yaml
Falling back to patching base and 3-way merge...
Auto-merging templates/test/ci/prow-workload-identity/kustomization.yaml
CONFLICT (content): Merge conflict in templates/test/ci/prow-workload-identity/kustomization.yaml
Auto-merging templates/test/ci/prow-edgezone/kustomization.yaml
CONFLICT (content): Merge conflict in templates/test/ci/prow-edgezone/kustomization.yaml
Removing templates/test/ci/patches/apiserver.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 Remove MixedProtocolLBService feature flag
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.15
/cherry-pick release-1.14
/hold cancel

As expected, the MEC e2e test failed, but everything else is green.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-infra-cherrypick-robot]

@mboersma: #4865 failed to apply on top of branch "release-1.14":

Applying: Bump e2e test k8s versions
Applying: Remove WindowsHostProcessContainers feature flag
Using index info to reconstruct a base tree...
A	test/e2e/data/infrastructure-azure/v1.14.5/cluster-template-prow.yaml
A	test/e2e/data/infrastructure-azure/v1.15.1/cluster-template-prow.yaml
Falling back to patching base and 3-way merge...
Applying: Remove MixedProtocolLBService feature flag
Using index info to reconstruct a base tree...
M	templates/test/ci/prow-edgezone/kustomization.yaml
M	templates/test/ci/prow-workload-identity/kustomization.yaml
Falling back to patching base and 3-way merge...
Auto-merging templates/test/ci/prow-workload-identity/kustomization.yaml
CONFLICT (content): Merge conflict in templates/test/ci/prow-workload-identity/kustomization.yaml
Auto-merging templates/test/ci/prow-edgezone/kustomization.yaml
CONFLICT (content): Merge conflict in templates/test/ci/prow-edgezone/kustomization.yaml
Removing templates/test/ci/patches/apiserver.yaml
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0003 Remove MixedProtocolLBService feature flag
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

In response to this:

/cherry-pick release-1.15
/cherry-pick release-1.14
/hold cancel

As expected, the MEC e2e test failed, but everything else is green.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.