CNTRLPLANE-2209: Add self-managed Azure HCP enhancement proposal #1904

bryan-cox · 2025-12-10T16:28:31Z

Define design for running HyperShift hosted control planes on
self-managed Azure infrastructure. Covers deployment workflow,
infrastructure prerequisites, and workload identity integration.

Fixes: https://issues.redhat.com/browse/CNTRLPLANE-2209

Signed-off-by: Bryan Cox brcox@redhat.com

openshift-ci · 2025-12-10T16:28:38Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2025-12-10T16:29:17Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign csrwng for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

enhancements/hypershift/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-ci-robot · 2025-12-10T16:53:40Z

@bryan-cox: This pull request references CNTRLPLANE-2209 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the task to target the "4.21.0" version, but no target version was set.

Details

In response to this:

Define design for running HyperShift hosted control planes on
self-managed Azure infrastructure. Covers deployment workflow,
infrastructure prerequisites, and workload identity integration.

Fixes: https://issues.redhat.com/browse/CNTRLPLANE-2209

Signed-off-by: Bryan Cox brcox@redhat.com

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

jsafrane · 2025-12-15T14:11:45Z

enhancements/hypershift/self-managed-azure.md

+deployments. While SNO could theoretically serve as a management cluster for
+HyperShift, this is not a target use case for self-managed Azure.
+
+### Implementation Details/Notes/Constraints


Here I would expect detailed info what components need to be updated for self-managed HCP on Azure and how. For me personally, the most important information would be that everyone in a hosted control plane who needs to talk to the Azure cloud must to use a token minter sidecar because of XYZ and how CPO tells its operands to do so.

You (HyperShift team) need to do a lot of work to support new hypershift install, hypershift create cluster azure etc., but there are no details about it.

The "Component Changes" section now provides the details you're looking for. The key points:

CPO tells operands to use token minter sidecar because hosted control plane components run in the management cluster but need to authenticate to Azure using the guest cluster's service accounts. The token-minter sidecar mints tokens from the guest cluster's API server (via kubeconfig) and writes them to a shared volume that the main container reads.

Why token minter is needed: Standard workload identity relies on projected service account tokens, but HCP components are in a different cluster than where their service accounts are defined. The token-minter bridges this gap.

HyperShift CLI work: The hypershift create cluster azure command already supports user-provided infrastructure. The hypershift create infra azure and hypershift create credentials azure commands are planned for Tech Preview to simplify setup.

The "Hosted Control Plane Components Requiring Token Minter Sidecar" table and "CSI Driver Token Minter Configuration" sections detail the specific components and how CPO configures them.

AI-assisted response via Claude Code

jsafrane · 2025-12-15T14:23:39Z

enhancements/hypershift/self-managed-azure.md

+## Proposal
+
+Self-managed Azure HyperShift extends the existing HyperShift architecture to
+support Azure as a platform where users manage all infrastructure themselves.
+The implementation leverages existing HyperShift patterns while adding
+Azure-specific infrastructure provisioning guidance and workload identity
+integration.
+
+The deployment consists of three main phases:
+
+1. **Azure Workload Identity Setup**: Create managed identities for OpenShift
+   components, configure the OIDC issuer, and establish federated credentials.
+
+2. **Management Cluster Setup**: Install the HyperShift operator on an existing
+   OpenShift cluster in Azure, optionally configure External DNS, and prepare
+   the cluster to host control planes.
+
+3. **Hosted Cluster Creation**: Provision Azure infrastructure for the hosted
+   cluster, deploy the control plane, create worker node VMs, and integrate
+   workload identities.


Copying from the template:

Enumerate all of the proposed changes at a high level, including all of the components that need to be modified and how they will be different. Include the reason for each choice in the design and implementation that is proposed here.

So, what components need to be modified?

I can see there is a list of components bellow now.

bryan-cox · 2025-12-15T14:52:59Z

@jsafrane Thank you for the feedback on component details. I've updated the proposal to address your comments:

Re: Comment at line 134 (Component list)
Added a new "Component Changes" subsection that enumerates:

HyperShift Repository Components - CLI commands (hypershift install, hypershift create cluster azure, hypershift create infra azure, hypershift create credentials azure), CPO, HyperShift Operator, and NodePool Controller with their specific modifications
Hosted Control Plane Components Requiring Token Minter Sidecar - Table listing each component (CCM, CSI drivers, Image Registry, Ingress, CAP/CAPZ) and their Azure API usage

Re: Comment at line 284 (Token minter sidecar details)
Added detailed "Azure Workload Identity Architecture" section explaining:

How the OIDC issuer is set up
Token flow diagram showing how control plane pods authenticate to Azure
CPO token minter configuration details (service account annotations, projected token volumes, environment variables)

The enhancement now documents how CPO configures operands to use workload identity federation for Azure API access.

AI-assisted response via Claude Code

bryan-cox · 2025-12-15T19:59:33Z

/test ?

openshift-ci · 2025-12-15T19:59:36Z

@bryan-cox: The following commands are available to trigger required jobs:

/test markdownlint

Use /test all to run all jobs.

Details

In response to this:

/test ?

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

bryan-cox · 2025-12-15T19:59:50Z

/test markdownlint

bryan-cox · 2025-12-15T20:35:45Z

/test markdownlint

bryan-cox · 2025-12-15T21:11:30Z

/test markdownlint

openshift-ci · 2025-12-15T21:27:26Z

@bryan-cox: all tests passed!

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

RomanBednar · 2025-12-16T12:37:41Z

enhancements/hypershift/self-managed-azure.md

+   - `hypershift create infra azure`: New infrastructure provisioning command
+     for creating Azure resources (VNets, subnets, NSGs, storage accounts).
+   - `hypershift create credentials azure`: New command for generating workload
+     identity credentials and federated credential configurations.


I haven't seen these in docs yet - is this supposed to simplify OIDC and identities setup? Is this aimed for TechPreview?

The hypershift create infra azure and hypershift create credentials azure commands are planned for Tech Preview to simplify the OIDC and workload identity setup. For Dev Preview, users will follow the manual setup documented at https://hypershift.pages.dev/how-to/azure/self-managed-azure-index/. The goal for Tech Preview is to provide a more streamlined experience similar to what we offer for AWS.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T12:46:10Z

enhancements/hypershift/self-managed-azure.md

+1. Projects a Kubernetes service account token to a well-known file path
+2. Configures Azure SDK environment variables for workload identity
+   authentication
+3. Enables components to obtain Azure access tokens without long-lived


Just leaving a note here: Azure CSI drivers still don't use "real" short-term credential setup (SMB), and we don't have ETA yet: kubernetes-sigs/azurefile-csi-driver#1737 (comment)

But it should not block this enhancement, and I'd expect this to be rather hidden fix without behavior change - but something we might want to focus on when testing.

Thanks for the note. I've added a callout in the CSI Driver section about this upstream limitation (kubernetes-sigs/azurefile-csi-driver#1737). This is good to keep in mind for testing but as you noted, shouldn't block the enhancement and will be transparent to users when upstream support is available.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T14:51:19Z

enhancements/hypershift/self-managed-azure.md

+- A projected service account token volume
+- Environment variables: `AZURE_CLIENT_ID`, `AZURE_TENANT_ID`,
+  `AZURE_FEDERATED_TOKEN_FILE`
+- The appropriate managed identity client ID for its Azure RBAC permissions


Suggested change

- The appropriate managed identity client ID for its Azure RBAC permissions

- The appropriate workload identity client ID for its Azure RBAC permissions

Done. Changed "managed identity" to "workload identity" for consistency.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T14:53:03Z

enhancements/hypershift/self-managed-azure.md

+
+| Environment Variable | Value |
+|---------------------|-------|
+| `AZURE_CLIENT_ID` | Managed identity client ID from `HostedCluster.Spec.Platform.Azure.WorkloadIdentities.<component>.ClientID` |


Suggested change

| `AZURE_CLIENT_ID` | Managed identity client ID from `HostedCluster.Spec.Platform.Azure.WorkloadIdentities.<component>.ClientID` |

| `AZURE_CLIENT_ID` | Workload identity client ID from `HostedCluster.Spec.Platform.Azure.WorkloadIdentities.<component>.ClientID` |

Done. Changed "Managed identity" to "Workload identity" for consistency.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T14:56:48Z

enhancements/hypershift/self-managed-azure.md

+   - Creates storage accounts for OIDC and image registry
+   - Creates managed identities for OpenShift components
+
+2. The platform engineer installs the OpenShift management cluster on Azure


Would be nice to have a statement on what type of management clusters are we expecting to support. So far we've been testing with standalone Azure OpenShift without Workload Idenitity - is that the only expected configuration we'll be supporting?

Done. Added a statement clarifying the supported management cluster configuration. For Dev Preview, the supported configuration is a standalone OpenShift cluster on Azure with Workload Identity (backed by federated managed identities).

AI-assisted response via Claude Code

Correction to my previous reply: The supported management cluster configuration is a standalone OpenShift cluster, which can run on Azure or AWS. Updated the document to reflect this.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T15:07:41Z

enhancements/hypershift/self-managed-azure.md

+3. **Workload Identity Integration**: Azure Workload Identity Federation is used
+   for secure authentication, requiring:
+   - OIDC issuer configuration
+   - Managed identities for each OpenShift component


Suggested change

- Managed identities for each OpenShift component

- Workload identities for each OpenShift component

Done. Changed "Managed identities" to "Workload identities" for consistency.

AI-assisted response via Claude Code

RomanBednar · 2025-12-16T15:09:40Z

enhancements/hypershift/self-managed-azure.md

+
+- Azure subscription for CI/CD testing with appropriate quotas
+- Test Azure infrastructure (VNets, storage accounts, DNS zones) for e2e tests
+- Integration with existing HyperShift CI infrastructure


Azure Graph API access is required.

Done. Added Azure Graph API access requirement to the Infrastructure Needed section.

AI-assisted response via Claude Code

Define design for running HyperShift hosted control planes on self-managed Azure infrastructure. Covers deployment workflow, infrastructure prerequisites, and workload identity integration. Fixes: https://issues.redhat.com/browse/CNTRLPLANE-2209 Signed-off-by: Bryan Cox <brcox@redhat.com>

openshift-ci bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 10, 2025

bryan-cox force-pushed the CNTRLPLANE-2209 branch from a39bad9 to 15d621e Compare December 10, 2025 16:45

bryan-cox changed the title ~~docs(hypershift): add self-managed Azure enhancement proposal~~ feature(hypershift): add self-managed Azure enhancement proposal Dec 10, 2025

bryan-cox changed the title ~~feature(hypershift): add self-managed Azure enhancement proposal~~ CNTRLPLANE-2209: Add self-managed Azure HCP enhancement proposal Dec 10, 2025

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Dec 10, 2025

bryan-cox force-pushed the CNTRLPLANE-2209 branch from 15d621e to 9aa012f Compare December 10, 2025 16:53

jsafrane reviewed Dec 15, 2025

View reviewed changes

bryan-cox force-pushed the CNTRLPLANE-2209 branch from 9aa012f to 2e370aa Compare December 15, 2025 14:51

bryan-cox force-pushed the CNTRLPLANE-2209 branch 6 times, most recently from c4589ea to 64b90d5 Compare December 15, 2025 17:44

bryan-cox force-pushed the CNTRLPLANE-2209 branch 2 times, most recently from d86d881 to 0db22b3 Compare December 15, 2025 20:35

bryan-cox force-pushed the CNTRLPLANE-2209 branch from 0db22b3 to bd19190 Compare December 15, 2025 21:11

RomanBednar reviewed Dec 16, 2025

View reviewed changes

bryan-cox force-pushed the CNTRLPLANE-2209 branch from bd19190 to 15a3f97 Compare December 17, 2025 12:38

bryan-cox force-pushed the CNTRLPLANE-2209 branch from 15a3f97 to b7691c5 Compare December 17, 2025 12:41

	- The appropriate managed identity client ID for its Azure RBAC permissions
	- The appropriate workload identity client ID for its Azure RBAC permissions

	\| `AZURE_CLIENT_ID` \| Managed identity client ID from `HostedCluster.Spec.Platform.Azure.WorkloadIdentities.<component>.ClientID` \|
	\| `AZURE_CLIENT_ID` \| Workload identity client ID from `HostedCluster.Spec.Platform.Azure.WorkloadIdentities.<component>.ClientID` \|

	- Managed identities for each OpenShift component
	- Workload identities for each OpenShift component

CNTRLPLANE-2209: Add self-managed Azure HCP enhancement proposal #1904

Are you sure you want to change the base?

CNTRLPLANE-2209: Add self-managed Azure HCP enhancement proposal #1904

Uh oh!

Conversation

bryan-cox commented Dec 10, 2025

Uh oh!

openshift-ci bot commented Dec 10, 2025

Uh oh!

openshift-ci bot commented Dec 10, 2025

Uh oh!

openshift-ci-robot commented Dec 10, 2025 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jsafrane Dec 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bryan-cox commented Dec 15, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

bryan-cox commented Dec 15, 2025

Uh oh!

openshift-ci bot commented Dec 15, 2025

Uh oh!

bryan-cox commented Dec 15, 2025

Uh oh!

bryan-cox commented Dec 15, 2025

Uh oh!

bryan-cox commented Dec 15, 2025

Uh oh!

openshift-ci bot commented Dec 15, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

openshift-ci-robot commented Dec 10, 2025 •

edited by openshift-ci bot

Loading

jsafrane Dec 16, 2025 •

edited

Loading

bryan-cox commented Dec 15, 2025 •

edited

Loading