CNTRLPLANE-3308: deps: bump k8s.io 0.34 → 0.35 and openshift/api

[muraee]

Summary

• Bump k8s.io/* from v0.34.3 to v0.35.1
• Bump github.com/openshift/api to 3c6b218b (openshift/api#2786) to pick up the ObservedRevisionGeneration field on ClusterAPIStatus
• Bump github.com/openshift/client-go to a19e917 (compatible with new API)
• Bump karpenter forks to versions built against k8s 0.35

Code fixes for API changes

• MustBaseEnvSet: removed bool param in k8s 0.35 (support/validations/authentication.go, control-plane-operator/.../auth.go)
• ClusterImagePolicy moved from config/v1alpha1 to config/v1 (hypershift-operator/controllers/nodepool/config.go)
• NodeSelectorRequirementWithMinValues no longer embeds corev1.NodeSelectorRequirement (test/e2e/karpenter_test.go)
• Removed etcd/tests/v3 dependency to eliminate olekukonko/tablewriter v0.x/v1.x conflict (etcdctl uses v0.x API, karpenter requires v1.x)

Why

The k8s 0.35 bump is required by the latest openshift/api which adds the ObservedRevisionGeneration field. This field is needed by PR #7996 to properly wait for the Cluster CAPI Operator to acknowledge unmanaged CRDs during hypershift install.

Test plan

• make build passes
• make test passes (all unit tests)
• go vet passes
• make update succeeds

🤖 Generated with Claude Code

Summary by CodeRabbit

• Chores
  • Updated Go toolchain to a newer patch release (build images and tooling aligned).
  • Refreshed core and indirect dependencies across the Kubernetes and related ecosystems.
  • Adjusted code verification tooling configuration to refine files excluded from checks.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[coderabbitai]

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

• @coderabbitai resume to resume automatic reviews.
• @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

• ▶️ Resume reviews
• 🔍 Trigger review

📝 Walkthrough

Walkthrough

The Makefile verify-codespell target’s codespell --skip list was adjusted to add ./api/go.sum while retaining existing skips such as ./go.sum, ./hack/workspace/go.work.sum, and other prior patterns. api/go.mod updates the Go toolchain directive from go 1.25.3 to go 1.25.7 and upgrades multiple direct and indirect dependencies (notably Kubernetes/OpenShift-related modules, go-openapi-related modules, and various indirects), plus updated replace directives to newer OpenShift pseudo-versions. Dockerfile.github-actions-runner updates the GO_VERSION build ARG from 1.25.3 to 1.25.7.

---

Important

Pre-merge checks failed

Please resolve all errors before merging. Addressing warnings is optional.

❌ Failed checks (1 error)

Check name	Status	Explanation	Resolution
Ote Binary Stdout Contract	❌ Error	PR introduces fmt.Println() and fmt.Printf() calls in TestE2EV2() function that write directly to stdout before RunSpecs() is called, violating OTE Binary Stdout Contract.	Replace fmt.Print*/Printf calls with fmt.Fprintf(os.Stderr, ...) or GinkgoWriter to redirect output from stdout to stderr.

✅ Passed checks (9 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly and specifically summarizes the main changes: bumping Kubernetes and OpenShift API dependencies.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Stable And Deterministic Test Names	✅ Passed	PR modifies Makefile, Dockerfile, and api/go.mod without introducing Ginkgo tests with dynamic test names.
Test Structure And Quality	✅ Passed	The PR does not introduce Ginkgo-style tests using Describe/Context/It blocks; it uses standard Go testing with t.Run() subtests and Gomega matchers. Since no Ginkgo test code is present, this check is not applicable.
Microshift Test Compatibility	✅ Passed	This PR is exclusively a dependency update bump with associated code changes to accommodate API compatibility changes. No new Ginkgo e2e tests are being added to the codebase.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	PR contains only dependency updates and modifications to existing tests, not new Ginkgo e2e test additions.
Topology-Aware Scheduling Compatibility	✅ Passed	PR contains only dependency upgrades and build configuration changes; no pod scheduling constraints incompatible with SNO/Two-Node/HyperShift topologies detected.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No new Ginkgo e2e tests are added in this PR; only existing test code is modified to handle API changes.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci-robot]

@muraee: This pull request references CNTRLPLANE-3308 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

Summary

• Bump k8s.io/* from v0.34.3 to v0.35.1
• Bump github.com/openshift/api to 3c6b218b (openshift/api#2786) to pick up the ObservedRevisionGeneration field on ClusterAPIStatus
• Bump github.com/openshift/client-go to a19e917 (compatible with new API)
• Bump karpenter forks to versions built against k8s 0.35

Code fixes for API changes

• MustBaseEnvSet: removed bool param in k8s 0.35 (support/validations/authentication.go, control-plane-operator/.../auth.go)
• ClusterImagePolicy moved from config/v1alpha1 to config/v1 (hypershift-operator/controllers/nodepool/config.go)
• NodeSelectorRequirementWithMinValues no longer embeds corev1.NodeSelectorRequirement (test/e2e/karpenter_test.go)
• Removed etcd/tests/v3 dependency to eliminate olekukonko/tablewriter v0.x/v1.x conflict (etcdctl uses v0.x API, karpenter requires v1.x)

Why

The k8s 0.35 bump is required by the latest openshift/api which adds the ObservedRevisionGeneration field. This field is needed by PR #7996 to properly wait for the Cluster CAPI Operator to acknowledge unmanaged CRDs during hypershift install.

Test plan

• make build passes
• make test passes (all unit tests)
• go vet passes
• make update succeeds

🤖 Generated with Claude Code

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[muraee]

/test "ci/prow/security"

[muraee]

/retest

[muraee]

/test e2e-aws
/test e2e-aks

[muraee]

/test e2e-aws
/test e2e-aks

[muraee]

/test e2e-aws
/test e2e-aks

[hypershift-jira-solve-ci]

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2049034916259172352 | Cost: $2.15918875 | Failed step: hypershift-azure-run-e2e

View full analysis report

---

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

[muraee]

/retest

[muraee]

/test e2e-aws
/test e2e-aks

[bryan-cox]

/lgtm

[bryan-cox]

/verified by e2e & ut

[openshift-ci-robot]

@bryan-cox: This PR has been marked as verified by e2e & ut.

Details

In response to this:

/verified by e2e & ut

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-merge-bot]

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks-4-22
/test e2e-aws-4-22
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

[openshift-ci]

@muraee: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aks-4-22	375a8c9	link	true	/test e2e-aks-4-22
ci/prow/e2e-azure-self-managed	375a8c9	link	true	/test e2e-azure-self-managed
ci/prow/e2e-kubevirt-aws-ovn-reduced	375a8c9	link	true	/test e2e-kubevirt-aws-ovn-reduced
ci/prow/e2e-aws-upgrade-hypershift-operator	375a8c9	link	true	/test e2e-aws-upgrade-hypershift-operator
ci/prow/e2e-aws-4-22	375a8c9	link	true	/test e2e-aws-4-22
ci/prow/e2e-v2-aws	375a8c9	link	true	/test e2e-v2-aws
ci/prow/okd-scos-images	08add78	link	true	/test okd-scos-images
ci/prow/images	08add78	link	true	/test images

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[hypershift-jira-solve-ci]

Now I have a complete picture. Let me produce the final report:

Test Failure Analysis Complete

Job Information

• Prow Job: Envtest OCP API Validation (GitHub Actions)
• Build ID: 25110153446
• PR: #8286 — CNTRLPLANE-3308: deps: bump k8s.io 0.34 → 0.35 and openshift/api
• Failed Jobs:
  • Envtest OCP (K8s 1.33.2) — job 73581779899
  • Envtest OCP (K8s 1.35.1) — job 73581779890
  • Conclusion — job 73583306249 (aggregate gate)

Test Failure Analysis

Error

[FAIL] CRD Installation [It] should install all CRDs for feature set "Default"
  test/envtest/generator.go:262

[FAILED] Timed out after 30.001s.
  CRD clusters.cluster.x-k8s.io should be fully removed          (K8s 1.35.1)
  CRD awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io should be fully removed  (K8s 1.33.2)

588 Passed | 1 Failed | 0 Pending | 0 Skipped

Summary

The GenerateCRDInstallTest("Default") test installs all 69 HyperShift/CAPI CRDs, then uninstalls them and polls each CRD with a 30-second timeout to confirm deletion. The CRD deletion request succeeds (no error from envtest.UninstallCRDs), but the Kubernetes API server takes longer than 30 seconds to fully remove certain CAPI CRDs (the specific CRD varies non-deterministically between runs). This is a pre-existing flaky test that also fails on the main branch (2/30 = 6.7% flake rate), but the k8s.io v0.34→v0.35 dependency bump in this PR increases the flake rate to approximately 26% (6/23 runs fail). K8s 1.35.1 is the most affected version (fails in 5 of 6 failing runs), with K8s 1.33.2 and 1.34.1 also occasionally affected.

Root Cause

The test at test/envtest/generator.go:262 uses a 30-second timeout for CRD deletion verification:

Eventually(func() bool {
    err := k8sClient.Get(ctx, key, &apiextensionsv1.CustomResourceDefinition{})
    return apierrors.IsNotFound(err)
}, "30s", "1s").Should(BeTrue(), fmt.Sprintf("CRD %s should be fully removed", crd.Name))

After envtest.UninstallCRDs() sends deletion requests for all 69 CRDs, the envtest API server processes these deletions asynchronously. The CRD finalizer (customresourcecleanup.apiextensions.k8s.io) must complete before each CRD is fully removed. Under resource pressure (69 simultaneous CRD deletions on a single-node envtest server), some CRDs take 30-35 seconds to be garbage-collected, exceeding the timeout.

Why the PR worsens the flake rate (~4x increase):

1. k8s.io/apiextensions-apiserver v0.34→v0.35 type changes: The client-side CRD type definitions in v0.35 include structural changes (e.g., ObservedRevisionGeneration field in ClusterAPIStatus from OCPCLOUD-3359: Add component names, manifestSubstitutions, and observedGeneration to CAPI revisions api#2786). When k8sClient.Get() deserializes the CRD from the envtest API server, the v0.35 client may take marginally longer to process the response, tightening the already-thin timing margin.

2. CRD manifest changes: The Default feature set CRDs (hostedclusters, hostedcontrolplanes) grew slightly (+800 bytes each) from the openshift/api bump, and new TLSAdherence feature gate manifests were added (14K lines total). While TLSAdherence is not in the Default feature set, the code generation and loading adds I/O overhead to the test environment.

3. Version skew: controller-runtime is pinned to v0.19.7 (designed for k8s.io v0.31) via a replace directive, but is now operating against k8s.io v0.35 client types — a 4-version skew (vs 3-version on main). This increases the likelihood of subtle serialization/deserialization timing differences.

Why it's non-deterministic: The CRD that times out varies across runs (clusters.cluster.x-k8s.io, clusterclasses.cluster.x-k8s.io, machinedrainrules.cluster.x-k8s.io, awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io, clusterresourcesets.addons.cluster.x-k8s.io) because CRD finalization order depends on API server scheduling, which varies with GitHub Actions runner load.

Recommendations

1. Increase CRD deletion timeout from 30s to 60s in test/envtest/generator.go:262. This is the simplest fix and addresses both the PR-introduced regression and the pre-existing main-branch flake. The deletion consistently completes within ~35 seconds, so 60s provides adequate headroom.

2. Alternative: delete CRDs in smaller batches — instead of uninstalling all 69 CRDs simultaneously, batch them in groups of 10-15 to reduce API server contention during finalization.

3. File a separate issue to track the underlying envtest CRD deletion flakiness on main (affects K8s 1.34.1 and 1.35.1 envtest versions).

4. Re-run the workflow — since this is a flaky test (passes 3 out of 4 times on this PR), a retry will likely pass. However, the increased flake rate should be addressed.

Evidence

Evidence	Detail
Test file	test/envtest/generator.go:262 — 30s timeout on Eventually() CRD deletion check
K8s 1.35.1 failing CRD	clusters.cluster.x-k8s.io (this run); varies across runs
K8s 1.33.2 failing CRD	awsclustercontrolleridentities.infrastructure.cluster.x-k8s.io
Test duration at failure	35.477s (K8s 1.33.2), 30.001s timeout hit (K8s 1.35.1)
PR branch flake rate	6/23 runs fail (26.1%) — mostly K8s 1.35.1, occasionally 1.33.2 and 1.34.1
Main branch flake rate	2/30 runs fail (6.7%) — same generator.go:262 timeout, same CRD type (CAPI)
Main branch failing CRDs	clusterresourcesets.addons.cluster.x-k8s.io (both main failures)
Passing PR runs	25108305374, 25102825490, 25051981101 — all 6 K8s versions pass
Dependency change	k8s.io/apiextensions-apiserver v0.34.3 → v0.35.1; controller-runtime remains v0.19.7 (replace)
CRD count	69 CRDs installed/uninstalled simultaneously in the "Default" feature set test
New CRD manifests	TLSAdherence feature gate: +7091 lines (hostedclusters) + 6935 lines (hostedcontrolplanes)
Conclusion job	Pure aggregate gate — fails because envtest-ocp matrix has failures

---