CNTRLPLANE-3600: Bump Prow CI build root image to Go 1.26 by bryan-cox · Pull Request #8941 · openshift/hypershift

bryan-cox · 2026-07-06T17:42:34Z

What this PR does / why we need it:

Bumps the Prow CI build root image from golang-1.25 to golang-1.26.

The k8s v0.36.2 dependency bump in #8695 sets go 1.26.0 in go.mod. PR #8888 already updated the GHA runner image to Go 1.26.4, but Prow e2e jobs still fail because the build root image has Go 1.25.8:

go: go.mod requires go >= 1.26.0 (running go 1.25.8; GOTOOLCHAIN=local)

Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3600

Special notes for your reviewer:

Prerequisite for #8695 (k8s bump). The golang-1.26 image already exists in the CI registry.

Checklist:

Subject and description added to both, commit and PR.
Relevant issues have been referenced.
This change includes docs.
This change includes unit tests.

Summary by CodeRabbit

Chores
- Updated the build job environment to use a newer Go 1.26-based image, keeping release builds on a current platform.

The k8s v0.36.2 dependency bump requires Go 1.26 (go.mod has go 1.26.0). Update the ci-operator build root image from golang-1.25 to golang-1.26 so Prow jobs can compile. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-merge-bot · 2026-07-06T17:42:37Z

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

openshift-ci-robot · 2026-07-06T17:42:38Z

@bryan-cox: This pull request references CNTRLPLANE-3600 which is a valid jira issue.

Details

In response to this:

What this PR does / why we need it:

Bumps the Prow CI build root image from golang-1.25 to golang-1.26.

The k8s v0.36.2 dependency bump in #8695 sets go 1.26.0 in go.mod. PR #8888 already updated the GHA runner image to Go 1.26.4, but Prow e2e jobs still fail because the build root image has Go 1.25.8:
go: go.mod requires go >= 1.26.0 (running go 1.25.8; GOTOOLCHAIN=local)
Which issue(s) this PR fixes:

Fixes CNTRLPLANE-3600

Special notes for your reviewer:

Prerequisite for #8695 (k8s bump). The golang-1.26 image already exists in the CI registry.

Checklist:

Subject and description added to both, commit and PR.

Relevant issues have been referenced.

This change includes docs.

This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

bryan-cox · 2026-07-06T17:44:44Z

/area ci-tooling

openshift-ci · 2026-07-06T17:45:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details

Needs approval from an approver in each of these files:

~~OWNERS~~ [bryan-cox]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-07-06T17:48:37Z

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: a326836a-e7bd-40e7-bb23-7c2e52c7c5dd

📥 Commits

Reviewing files that changed from the base of the PR and between dda6055 and db8bc73.

📒 Files selected for processing (1)

.ci-operator.yaml

📝 Walkthrough

Walkthrough

This change updates the tag field for build_root_image in .ci-operator.yaml, switching from the RHEL 9 OpenShift 4.23 Go 1.25 image to the corresponding Go 1.26 image.

Possibly related PRs

openshift/hypershift#8888: Also updates the Go toolchain version from 1.25 to 1.26, but in the GitHub Actions runner image configuration.

🚥 Pre-merge checks | ✅ 11

✅ Passed checks (11 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly matches the main change: bumping the Prow CI build root image to Go 1.26.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	PR only updates .ci-operator.yaml build_root_image tag; no Ginkgo tests or titles were added/changed, so no unstable names to flag.
Test Structure And Quality	✅ Passed	Only .ci-operator.yaml changed; no Ginkgo test code was modified, so the test-quality check is not applicable.
Topology-Aware Scheduling Compatibility	✅ Passed	Only .ci-operator.yaml changed, bumping the build_root_image tag from Go 1.25 to 1.26; no manifests, controllers, or scheduling constraints were modified.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	Only .ci-operator.yaml changed to bump the build_root_image tag; no Ginkgo e2e tests were added or modified.
No-Weak-Crypto	✅ Passed	PR only changes the CI build_root_image tag from Go 1.25 to 1.26; no crypto code, weak algorithms, or secret comparisons are touched.
Container-Privileges	✅ Passed	PR only bumps the build_root_image tag from golang-1.25 to golang-1.26; no privileged, hostPID/Network/IPC, or allowPrivilegeEscalation fields are present or changed.
No-Sensitive-Data-In-Logs	✅ Passed	Only .ci-operator.yaml build root image tag changed; no logging or sensitive-data patterns were introduced.

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

_{Comment @coderabbitai help to get the list of available commands.}

codecov · 2026-07-06T17:50:37Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 43.45%. Comparing base (dda6055) to head (db8bc73).
⚠️ Report is 18 commits behind head on main.

Additional details and impacted files

@@           Coverage Diff           @@
##             main    #8941   +/-   ##
=======================================
  Coverage   43.45%   43.45%           
=======================================
  Files         771      771           
  Lines       95718    95718           
=======================================
  Hits        41597    41597           
  Misses      51234    51234           
  Partials     2887     2887

Flag	Coverage Δ
cmd-support	`37.12% <ø> (ø)`
cpo-hostedcontrolplane	`45.21% <ø> (ø)`
cpo-other	`45.10% <ø> (ø)`
hypershift-operator	`53.65% <ø> (ø)`
other	`32.08% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

csrwng · 2026-07-06T18:47:12Z

/lgtm

openshift-merge-bot · 2026-07-06T18:47:27Z

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-v2-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws
/test e2e-v2-gke

bryan-cox · 2026-07-06T18:51:25Z

/verified by e2e

openshift-ci-robot · 2026-07-06T18:51:37Z

@bryan-cox: This PR has been marked as verified by e2e.

Details

In response to this:

/verified by e2e

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

openshift-merge-bot · 2026-07-06T20:40:35Z

/retest-required

Remaining retests: 0 against base HEAD dda6055 and 2 for PR HEAD db8bc73 in total

hypershift-jira-solve-ci · 2026-07-06T21:34:27Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2074203787446194176 | Cost: $3.2893604999999986 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

cwbotbot · 2026-07-07T09:04:51Z

Test Results

e2e-aws

Status: ❌ FAIL
Started: 2026-07-09T17:19:46Z
View Job
View Job History

Failed Tests

Total failed tests: 26

TestAutoscaling
TestAutoscaling/ValidateHostedCluster
TestCreateCluster
TestCreateCluster/ValidateHostedCluster
TestCreateClusterCustomConfig

... and 21 more failed tests

e2e-aks

Status: ✅ PASS
Started: 2026-07-06T18:48:24Z
View Job
View Job History

JoelSpeed · 2026-07-07T09:25:46Z

/retest

openshift-merge-bot · 2026-07-07T21:19:22Z

/retest-required

Remaining retests: 0 against base HEAD a627c48 and 1 for PR HEAD db8bc73 in total

openshift-merge-bot · 2026-07-07T23:17:34Z

/retest-required

Remaining retests: 0 against base HEAD 1aca754 and 0 for PR HEAD db8bc73 in total

bryan-cox · 2026-07-07T23:56:54Z

/retest

openshift-merge-bot · 2026-07-08T01:17:07Z

/hold

Revision db8bc73 was retested 3 times: holding

hypershift-jira-solve-ci · 2026-07-08T02:11:06Z

AI Test Failure Analysis

Job: pull-ci-openshift-hypershift-main-e2e-aws | Build: 2074634357158449152 | Cost: $2.4040405 | Failed step: hypershift-aws-run-e2e-nested

View full analysis report

_{Generated by hypershift-analyze-e2e-failure post-step using Claude claude-opus-4-6}

bryan-cox · 2026-07-09T12:28:19Z

/retest

bryan-cox · 2026-07-09T17:19:33Z

/test e2e-aws

Issue with teardown flake

bryan-cox · 2026-07-11T01:29:26Z

/retest

bryan-cox · 2026-07-11T03:18:12Z

/test e2e-aws

All failures have been different. Last one looks like an infra issue.

bryan-cox · 2026-07-13T14:21:50Z

/retest

clebs · 2026-07-13T17:39:59Z

/lgtm

clebs · 2026-07-13T17:40:06Z

/test e2e-aws

bryan-cox · 2026-07-14T16:57:04Z

/test e2e-aws

openshift-ci · 2026-07-14T20:18:46Z

@bryan-cox: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-aws	`db8bc73`	link	true	`/test e2e-aws`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

hypershift-jira-solve-ci · 2026-07-15T07:22:38Z

The analyzer timed out waiting for artifacts. Now I have all the information needed. Let me produce the final report.

Test Failure Analysis Complete

Job Information

Prow Job: pull-ci-openshift-hypershift-main-e2e-aws
Build ID: 2077074928846245888
Target: e2e-aws
PR: CNTRLPLANE-3600: Bump Prow CI build root image to Go 1.26 #8941 — CNTRLPLANE-3600: Bump Prow CI build root image to Go 1.26
Failed Test: TestCreateCluster/Main/EnsureGlobalPullSecret/When_management-cluster_hostedCluster.Spec.PullSecret_is_updated_in-place_it_should_propagate_to_guest_without_rollout

Test Failure Analysis

Error

failed to wait for DaemonSet global-pull-secret-syncer to be ready: context deadline exceeded

DaemonSet global-pull-secret-syncer stuck at 2/3 pods ready for ~1265 seconds until timeout.

Summary

The EnsureGlobalPullSecret subtest patches the management-cluster pull secret in-place and waits for propagation to the guest cluster. As part of this, it deploys a global-pull-secret-syncer DaemonSet and waits for all pods to become ready. The DaemonSet remained stuck at 2/3 pods ready (one pod never became ready on one of the 3 nodes), eventually hitting a context deadline after ~21 minutes. This is a known flaky test (~7% failure rate) unrelated to the Go 1.26 build root bump in this PR. The PR only modifies .ci-operator.yaml to change the CI test runner image — the operator binaries deployed in the cluster are still compiled with Go 1.25 from the Dockerfiles.

Root Cause

Known flaky test — not caused by PR changes.

The global-pull-secret-syncer DaemonSet failing to reach 3/3 ready pods is a known intermittent issue tracked under:

OCPBUGS-98464 (fixed in PR OCPBUGS-98464: handle stale DaemonSet in GlobalPullSecret test #8990, merged 2026-07-14): Addresses a stale kubelet-config-verifier DaemonSet cleanup issue.
OCPBUGS-98465 (PR OCPBUGS-98465: prevent DS crash in pull secret verifier and add propagation diagnostics #8991, open/on hold): Adds diagnostic logging to investigate the root cause. Leading theory is guest KAS (kube-apiserver) i/o timeout during the pull secret propagation window, preventing the DaemonSet rolling update from completing on all nodes.

The exact same failure (2/3 pods stuck, context deadline exceeded after ~1200s) has been observed on other PRs that do not involve Go version changes (e.g., build 2076767540909969408 on PR #8990).

Why this is unrelated to the Go 1.26 bump:

PR CNTRLPLANE-3600: Bump Prow CI build root image to Go 1.26 #8941 modifies only .ci-operator.yaml, changing the build root image from golang-1.25 to golang-1.26.
The build root image is used only for CI test/lint steps (test runner environment).
All four Dockerfiles (Dockerfile, Dockerfile.control-plane, Dockerfile.e2e, Dockerfile.dev) continue to use golang-1.25 base images — the actual hypershift-operator, control-plane-operator, and e2e test binaries deployed in the cluster are compiled with Go 1.25.
The software under test is byte-for-byte identical to what runs without this PR.

Recommendations

Retry the job — This is a flaky test with ~7% failure rate. A re-run is expected to pass.
No code changes needed in this PR — The failure is not caused by the Go 1.26 build root bump.
Track OCPBUGS-98465 — The upstream fix (PR OCPBUGS-98465: prevent DS crash in pull secret verifier and add propagation diagnostics #8991) adds diagnostic logging to identify why the global-pull-secret-syncer DaemonSet intermittently fails to become ready on all nodes. Once the root cause is identified, a follow-up fix will reduce the flake rate.

Evidence

Evidence	Detail
Failed test	`TestCreateCluster/Main/EnsureGlobalPullSecret/When_management-cluster_hostedCluster.Spec.PullSecret_is_updated_in-place_it_should_propagate_to_guest_without_rollout`
Error message	`failed to wait for DaemonSet global-pull-secret-syncer to be ready: context deadline exceeded`
DaemonSet status	Stuck at 2/3 pods ready for ~1265 seconds (logged 100+ times at `util.go:2290`)
Failure duration	1265.18s (~21 minutes) before timeout
PR changes	Single file: `.ci-operator.yaml` — build root image tag `golang-1.25` → `golang-1.26`
Build root (line 30)	`openshift/release:rhel-9-release-golang-1.26-openshift-4.23` (CI test runner only)
Operator images (lines 33-34)	`ocp_builder:rhel-9-golang-1.25-openshift-4.23` and `openshift_release:rhel-9-release-golang-1.25-openshift-4.23` (unchanged)
Known flake rate	~7% (1 in 14 runs), documented in PR #8991
Tracking bugs	OCPBUGS-98464 (fixed), OCPBUGS-98465 (open, on hold)
Identical failure on other PRs	Build `2076767540909969408` on PR #8990 (same 2/3-stuck symptom, no Go version change)
All other tests	619 passed, 30 skipped — only the `EnsureGlobalPullSecret` subtest failed

build: bump Prow CI build root image to Go 1.26

db8bc73

The k8s v0.36.2 dependency bump requires Go 1.26 (go.mod has go 1.26.0). Update the ci-operator build root image from golang-1.25 to golang-1.26 so Prow jobs can compile. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jul 6, 2026

openshift-ci Bot added the do-not-merge/needs-area label Jul 6, 2026

openshift-ci Bot requested review from clebs and csrwng July 6, 2026 17:44

openshift-ci Bot added area/ci-tooling Indicates the PR includes changes for CI or tooling and removed do-not-merge/needs-area labels Jul 6, 2026

openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 6, 2026

openshift-ci Bot assigned csrwng Jul 6, 2026

openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Jul 6, 2026

openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Jul 6, 2026

openshift-ci Bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jul 8, 2026

openshift-ci Bot assigned clebs Jul 13, 2026

Uh oh!

Conversation

bryan-cox commented Jul 6, 2026 • edited by coderabbitai Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Summary by CodeRabbit

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 6, 2026 • edited by openshift-ci Bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Checklist:

Uh oh!

bryan-cox commented Jul 6, 2026

Uh oh!

openshift-ci Bot commented Jul 6, 2026

Uh oh!

coderabbitai Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Walkthrough

Uh oh!

codecov Bot commented Jul 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

csrwng commented Jul 6, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

bryan-cox commented Jul 6, 2026

Uh oh!

openshift-ci-robot commented Jul 6, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 6, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jul 6, 2026

AI Test Failure Analysis

Uh oh!

cwbotbot commented Jul 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Test Results

e2e-aws

e2e-aks

Uh oh!

JoelSpeed commented Jul 7, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 7, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 7, 2026

Uh oh!

bryan-cox commented Jul 7, 2026

Uh oh!

openshift-merge-bot Bot commented Jul 8, 2026

Uh oh!

hypershift-jira-solve-ci Bot commented Jul 8, 2026

AI Test Failure Analysis

Uh oh!

bryan-cox commented Jul 9, 2026

Uh oh!

bryan-cox commented Jul 9, 2026

Uh oh!

bryan-cox commented Jul 11, 2026

Uh oh!

bryan-cox commented Jul 11, 2026

Uh oh!

bryan-cox commented Jul 13, 2026

Uh oh!

clebs commented Jul 13, 2026

Uh oh!

clebs commented Jul 13, 2026

Uh oh!

bryan-cox commented Jul 14, 2026

Uh oh!

openshift-ci Bot commented Jul 14, 2026

bryan-cox commented Jul 6, 2026 •

edited by coderabbitai Bot

Loading

openshift-ci-robot commented Jul 6, 2026 •

edited by openshift-ci Bot

Loading

coderabbitai Bot commented Jul 6, 2026 •

edited

Loading

codecov Bot commented Jul 6, 2026 •

edited

Loading

cwbotbot commented Jul 7, 2026 •

edited

Loading

hypershift-jira-solve-ci Bot commented Jul 15, 2026 •

edited by openshift-ci Bot

Loading