Skip to content

OCPBUGS-101964: [release-4.22] OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends - #9212

Merged
openshift-merge-bot[bot] merged 6 commits into
openshift:release-4.22from
bennerv:cherry-pick-OCPBUGS-98213-to-release-4.22
Aug 7, 2026
Merged

OCPBUGS-101964: [release-4.22] OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends#9212
openshift-merge-bot[bot] merged 6 commits into
openshift:release-4.22from
bennerv:cherry-pick-OCPBUGS-98213-to-release-4.22

Conversation

@bennerv

@bennerv bennerv commented Aug 3, 2026

Copy link
Copy Markdown
Member

Summary

Cherry-pick of PR #8971 to release-4.22 with manual conflict resolution and adaptation fixes.

  • Moves router component registration after all route-creating components (ignition-server, metrics-proxy) so the HAProxy config includes every route on the first reconcile pass
  • Adds router predicate on ARO-HCP to wait for expected routes before reconciling
  • Excludes ignition-server and ignition-server-proxy from implicit KAS dependency
  • Conditionally includes oauth-internal in ARO route predicate

Commits

  • 77e577f6c4: fix(control-plane-operator): ensure router component runs after route-creating components
  • c995ad02f6: fix(control-plane-operator): drop metrics-proxy from router WithDependencies
  • 2b9df60b2f: fix(control-plane-component): exclude ignition-server from implicit KAS dependency
  • 00e36f3406: fix(router): conditionally include oauth-internal in ARO route predicate
  • d565a6ed5b: fix(control-plane-component): exclude ignition-server-proxy from implicit KAS dependency

Fixes Applied

  • Resolved merge conflict in hostedcontrolplane_controller_test.go (test functions present on main but not on release-4.22)
  • Replaced support/netutil imports with support/util (netutil package does not exist on release-4.22)
  • Replaced gomock-generated mock with FakeReleaseProvider (mock file not generated on release-4.22)

References

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

OpenShift CI Bot and others added 6 commits August 3, 2026 14:52
…-creating components

Add a predicate to the router component that ensures all expected HCP
router routes exist and are ready before reconciling on ARO-HCP clusters.
This includes kube-apiserver-internal, konnectivity-server, oauth-internal,
ignition-server, and conditionally metrics-proxy when metrics forwarding
is enabled.

The router also declares explicit WithDependencies on ignition-server and
metrics-proxy components to give the informer cache more time to sync
after those components create their routes via server-side apply.

Add a TODO for introducing live reloading (like shared proxy) so the
router config is updated when routes change after the initial reconcile.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…dencies

The metrics-proxy component is conditional (only created when
MetricsForwarding.Mode=Forward), so declaring it as a hard dependency
causes a deadlock on platforms where it is never instantiated (e.g. GKE).
The router predicate already ensures metrics-proxy routes exist before
reconciling, making the explicit dependency unnecessary.

Update all 5 router fixture YAML files to match the actual declared
dependency (ignition-server only).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…AS dependency

The ignition-server only needs the management cluster KAS (via its
service account token), not the hosted cluster KAS. Including it in
the implicit KAS dependency creates a deadlock on private clusters
with KMS encryption (e.g. Azure ARO-HCP with private Key Vault):

  Router → ignition-server → KAS → Router (for KMS connectivity)

By excluding ignition-server from the KAS dependency, its deployment
and route are created independently, unblocking the router, which in
turn provides the network path KAS needs to reach the KMS endpoint.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The oauth-internal route is only created by the infra reconciler when
OAuth is enabled (HCPOAuthEnabled). When OAuth is disabled (e.g. OIDC
authentication), the route is never created, causing the ARO router
predicate to wait forever — a deadlock.

Move oauth-internal from the unconditional base list to a conditional
append gated on HCPOAuthEnabled, matching how the infra reconciler
decides whether to create the route.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…icit KAS dependency

The ignition-server-proxy component does not need the hosted KAS —
it only proxies to the ignition-server, which is already excluded.
Including it in the implicit KAS dependency creates the same deadlock
as ignition-server on private clusters with KMS encryption: the router
waits for the ignition-server-proxy Service to exist, but ignition-
server-proxy waits for KAS, which waits for the router.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Replace support/netutil imports with support/util (netutil package
  does not exist on release-4.22)
- Use FakeReleaseProvider instead of gomock-generated mock
  (mock file not generated on release-4.22)

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Aug 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bennerv: This pull request references Jira Issue OCPBUGS-98213, which is invalid:

  • expected the bug to target either version "4.22." or "openshift-4.22.", but it targets "5.0.0" instead
  • expected the bug to be in one of the following states: NEW, ASSIGNED, POST, but it is ON_QA instead
  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.
  • expected Jira Issue OCPBUGS-98213 to depend on a bug targeting a version in 5.0.0 and in one of the following states: MODIFIED, ON_QA, VERIFIED, but no dependents were found

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Cherry-pick of PR #8971 to release-4.22 with manual conflict resolution and adaptation fixes.

  • Moves router component registration after all route-creating components (ignition-server, metrics-proxy) so the HAProxy config includes every route on the first reconcile pass
  • Adds router predicate on ARO-HCP to wait for expected routes before reconciling
  • Excludes ignition-server and ignition-server-proxy from implicit KAS dependency
  • Conditionally includes oauth-internal in ARO route predicate

Commits

  • 77e577f6c4: fix(control-plane-operator): ensure router component runs after route-creating components
  • c995ad02f6: fix(control-plane-operator): drop metrics-proxy from router WithDependencies
  • 2b9df60b2f: fix(control-plane-component): exclude ignition-server from implicit KAS dependency
  • 00e36f3406: fix(router): conditionally include oauth-internal in ARO route predicate
  • d565a6ed5b: fix(control-plane-component): exclude ignition-server-proxy from implicit KAS dependency

Fixes Applied

  • Resolved merge conflict in hostedcontrolplane_controller_test.go (test functions present on main but not on release-4.22)
  • Replaced support/netutil imports with support/util (netutil package does not exist on release-4.22)
  • Replaced gomock-generated mock with FakeReleaseProvider (mock file not generated on release-4.22)

References

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai

coderabbitai Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Repository YAML (base), Central YAML (inherited)

Review profile: CHILL

Plan: Enterprise

Run ID: bf858a17-09f0-48e8-b0f8-db13d2dca3bd

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Comment @coderabbitai help to get the list of available commands.

@openshift-ci
openshift-ci Bot requested review from jparrill and sjenning August 3, 2026 19:03
@openshift-ci openshift-ci Bot added area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release and removed do-not-merge/needs-area labels Aug 3, 2026
@bennerv

bennerv commented Aug 3, 2026

Copy link
Copy Markdown
Member Author

/jira cherrypick OCPBUGS-98213

@openshift-ci-robot

Copy link
Copy Markdown

@bennerv: Jira Issue OCPBUGS-98213 has been cloned as Jira Issue OCPBUGS-101964. Will retitle bug to link to clone.
/retitle OCPBUGS-101964: [release-4.22] OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends

Details

In response to this:

/jira cherrypick OCPBUGS-98213

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci Bot changed the title [release-4.22] OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends OCPBUGS-101964: [release-4.22] OCPBUGS-98213: fix router component ordering to prevent missing HAProxy backends Aug 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bennerv: This pull request references Jira Issue OCPBUGS-101964, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

Summary

Cherry-pick of PR #8971 to release-4.22 with manual conflict resolution and adaptation fixes.

  • Moves router component registration after all route-creating components (ignition-server, metrics-proxy) so the HAProxy config includes every route on the first reconcile pass
  • Adds router predicate on ARO-HCP to wait for expected routes before reconciling
  • Excludes ignition-server and ignition-server-proxy from implicit KAS dependency
  • Conditionally includes oauth-internal in ARO route predicate

Commits

  • 77e577f6c4: fix(control-plane-operator): ensure router component runs after route-creating components
  • c995ad02f6: fix(control-plane-operator): drop metrics-proxy from router WithDependencies
  • 2b9df60b2f: fix(control-plane-component): exclude ignition-server from implicit KAS dependency
  • 00e36f3406: fix(router): conditionally include oauth-internal in ARO route predicate
  • d565a6ed5b: fix(control-plane-component): exclude ignition-server-proxy from implicit KAS dependency

Fixes Applied

  • Resolved merge conflict in hostedcontrolplane_controller_test.go (test functions present on main but not on release-4.22)
  • Replaced support/netutil imports with support/util (netutil package does not exist on release-4.22)
  • Replaced gomock-generated mock with FakeReleaseProvider (mock file not generated on release-4.22)

References

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@codecov

codecov Bot commented Aug 3, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 85.45455% with 8 lines in your changes missing coverage. Please review.
✅ Project coverage is 36.36%. Comparing base (41cbcdf) to head (829f432).
⚠️ Report is 4 commits behind head on release-4.22.

Files with missing lines Patch % Lines
...trollers/hostedcontrolplane/v2/router/component.go 85.18% 6 Missing and 2 partials ⚠️
Additional details and impacted files
@@               Coverage Diff                @@
##           release-4.22    #9212      +/-   ##
================================================
+ Coverage         36.34%   36.36%   +0.02%     
================================================
  Files               775      775              
  Lines             95049    95110      +61     
================================================
+ Hits              34545    34591      +46     
- Misses            57679    57692      +13     
- Partials           2825     2827       +2     
Files with missing lines Coverage Δ
...ostedcontrolplane/hostedcontrolplane_controller.go 36.78% <100.00%> (ø)
support/controlplane-component/status.go 72.28% <ø> (ø)
...trollers/hostedcontrolplane/v2/router/component.go 63.88% <85.18%> (+63.88%) ⬆️
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

func TestRouterComponentComesAfterRouteCreatingComponents(t *testing.T) {
t.Parallel()

reconciler := &HostedControlPlaneReconciler{

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

releaseinfo.NewMockedProvider.... isn't on release-4.22, hence why we just use a fake release provider here.

bennerv added a commit to bennerv/ARO-HCP that referenced this pull request Aug 3, 2026
Built from openshift/hypershift cherry-pick-OCPBUGS-98213-to-release-4.22
at commit 829f43241b (openshift/hypershift#9212).

Image: arohcpocpdev.azurecr.io/control-plane-operator@sha256:2a9a553057e74e9263829ad636213cad63f9d8ae91a104e0a3d593a0164ce294

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

@bryan-cox bryan-cox left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

/approve

@openshift-ci

openshift-ci Bot commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bennerv, bryan-cox

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci openshift-ci Bot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Aug 3, 2026
@bryan-cox

Copy link
Copy Markdown
Member

/jira refresh

@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This pull request references Jira Issue OCPBUGS-101964, which is invalid:

  • release note text must be set and not match the template OR release note type must be set to "Release Note Not Required". For more information you can reference the OpenShift Bug Process.

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bryan-cox

Copy link
Copy Markdown
Member

/jira refresh

@openshift-ci-robot openshift-ci-robot added the jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. label Aug 3, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@bryan-cox: This pull request references Jira Issue OCPBUGS-101964, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note type set to "Release Note Not Required"
  • dependent bug Jira Issue OCPBUGS-98213 is in the state Verified, which is one of the valid states (MODIFIED, ON_QA, VERIFIED)
  • dependent Jira Issue OCPBUGS-98213 targets the "5.0.0" version, which is one of the valid target versions: 5.0.0
  • bug has dependents
Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot openshift-ci-robot removed the jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. label Aug 3, 2026
bennerv added a commit to bennerv/ARO-HCP that referenced this pull request Aug 4, 2026
Built from openshift/hypershift cherry-pick-OCPBUGS-98213-to-release-4.22
at commit 829f43241b (openshift/hypershift#9212).

Image: arohcpocpdev.azurecr.io/control-plane-operator@sha256:2a9a553057e74e9263829ad636213cad63f9d8ae91a104e0a3d593a0164ce294

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
@twolff-gh

Copy link
Copy Markdown
Contributor

/verified by @twolff-gh
single rollout, complete HAProxy config from the first reconcile

CPO override test cluster showed 1 configmap create + 1 deployment create, no subsequent CPO updates, single replicaset

@openshift-ci-robot openshift-ci-robot added the verified Signifies that the PR passed pre-merge verification criteria label Aug 5, 2026
@openshift-ci-robot

Copy link
Copy Markdown

@twolff-gh: This PR has been marked as verified by @twolff-gh.

Details

In response to this:

/verified by @twolff-gh
single rollout, complete HAProxy config from the first reconcile

CPO override test cluster showed 1 configmap create + 1 deployment create, no subsequent CPO updates, single replicaset

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@twolff-gh

Copy link
Copy Markdown
Contributor

/lgtm

@openshift-ci openshift-ci Bot added the lgtm Indicates that a PR is ready to be merged. label Aug 5, 2026
@openshift-merge-bot

Copy link
Copy Markdown
Contributor

Scheduling tests matching the pipeline_run_if_changed or not excluded by pipeline_skip_if_only_changed parameters:
/test e2e-aks
/test e2e-aws
/test e2e-aws-upgrade-hypershift-operator
/test e2e-azure-self-managed
/test e2e-kubevirt-aws-ovn-reduced
/test e2e-v2-aws

@bennerv

bennerv commented Aug 6, 2026

Copy link
Copy Markdown
Member Author

/test e2e-v2-aws

@csrwng csrwng added the backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. label Aug 7, 2026
@openshift-ci

openshift-ci Bot commented Aug 7, 2026

Copy link
Copy Markdown
Contributor

@bennerv: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@openshift-merge-bot
openshift-merge-bot Bot merged commit 0cb524b into openshift:release-4.22 Aug 7, 2026
24 checks passed
@openshift-ci-robot

Copy link
Copy Markdown

@bennerv: Jira Issue Verification Checks: Jira Issue OCPBUGS-101964
✔️ This pull request was pre-merge verified.
✔️ All associated pull requests have merged.
✔️ All associated, merged pull requests were pre-merge verified.

Jira Issue OCPBUGS-101964 has been moved to the MODIFIED state and will move to the VERIFIED state when the change is available in an accepted nightly payload. 🕓

Details

In response to this:

Summary

Cherry-pick of PR #8971 to release-4.22 with manual conflict resolution and adaptation fixes.

  • Moves router component registration after all route-creating components (ignition-server, metrics-proxy) so the HAProxy config includes every route on the first reconcile pass
  • Adds router predicate on ARO-HCP to wait for expected routes before reconciling
  • Excludes ignition-server and ignition-server-proxy from implicit KAS dependency
  • Conditionally includes oauth-internal in ARO route predicate

Commits

  • 77e577f6c4: fix(control-plane-operator): ensure router component runs after route-creating components
  • c995ad02f6: fix(control-plane-operator): drop metrics-proxy from router WithDependencies
  • 2b9df60b2f: fix(control-plane-component): exclude ignition-server from implicit KAS dependency
  • 00e36f3406: fix(router): conditionally include oauth-internal in ARO route predicate
  • d565a6ed5b: fix(control-plane-component): exclude ignition-server-proxy from implicit KAS dependency

Fixes Applied

  • Resolved merge conflict in hostedcontrolplane_controller_test.go (test functions present on main but not on release-4.22)
  • Replaced support/netutil imports with support/util (netutil package does not exist on release-4.22)
  • Replaced gomock-generated mock with FakeReleaseProvider (mock file not generated on release-4.22)

References

🤖 Generated with Claude Code

Co-Authored-By: Claude Opus 4.6 (1M context) noreply@anthropic.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@bennerv
bennerv deleted the cherry-pick-OCPBUGS-98213-to-release-4.22 branch August 7, 2026 13:31
@bennerv

bennerv commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

/cherry-pick release-4.21

@openshift-cherrypick-robot

Copy link
Copy Markdown

@bennerv: #9212 failed to apply on top of branch "release-4.21":

Applying: fix(control-plane-operator): ensure router component runs after route-creating components
Using index info to reconstruct a base tree...
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
M	control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
Falling back to patching base and 3-way merge...
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller.go
Auto-merging control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
CONFLICT (content): Merge conflict in control-plane-operator/controllers/hostedcontrolplane/hostedcontrolplane_controller_test.go
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
hint: When you have resolved this problem, run "git am --continue".
hint: If you prefer to skip this patch, run "git am --skip" instead.
hint: To restore the original branch and stop patching, run "git am --abort".
hint: Disable this message with "git config set advice.mergeConflict false"
Patch failed at 0001 fix(control-plane-operator): ensure router component runs after route-creating components

Details

In response to this:

/cherry-pick release-4.21

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@bennerv

bennerv commented Aug 7, 2026

Copy link
Copy Markdown
Member Author

Cherry-pick to release-4.21 created in #9263 (manual conflict resolution required).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by an approver from all required OWNERS files. area/control-plane-operator Indicates the PR includes changes for the control plane operator - in an OCP release area/hypershift-operator Indicates the PR includes changes for the hypershift operator and API - outside an OCP release backport-risk-assessed Indicates a PR to a release branch has been evaluated and considered safe to accept. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. lgtm Indicates that a PR is ready to be merged. verified Signifies that the PR passed pre-merge verification criteria

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants