CNTRLPLANE-507: Add HCP finalizer to AWSEndpointService reconciler

[hypershift-jira-solve-ci]

What this PR does / why we need it:

Adds a finalizer on the HostedControlPlane resource from the AWSEndpointService reconciler to prevent HCP deletion before AWS PrivateLink resources are cleaned up.

Problem: When the CPO restarts during deletion of a SharedVPC cluster, the clientBuilder is uninitialized and the HCP (with its cross-account role ARNs) may already be deleted. This causes the reconciler to fail creating AWS clients, and after a 10-minute grace period the hypershift-operator force-removes the CPO finalizer — orphaning VPC endpoints, security groups, and DNS records in the shared VPC account.

Solution: The new HCP finalizer (hypershift.openshift.io/aws-private-link-endpoint-cleanup) follows the same pattern used by the Azure PLS controller:

• Adds the finalizer to the HCP during normal reconciliation
• When HCP deletion is detected, initializes AWS clients from the still-available HCP
• Cleans up each AWSEndpointService's AWS resources and removes CR finalizers
• Removes the HCP finalizer only after all AWSEndpointService CRs are cleaned up
• Extends the HCP watch handler (enqueueOnHCPChange) to also trigger reconciliation when an HCP is being deleted with the finalizer present

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-507

Special notes for your reviewer:

• This follows the same finalizer pattern already established by the Azure PLS controller
• The enqueueOnHCPChange handler (renamed from enqueueOnAccessChange) now triggers on both EndpointAccess changes and HCP deletions with the finalizer
• AWS client initialization during HCP deletion reuses the existing getAWSClient helper, sourcing credentials from the still-available HCP spec

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Always review AI generated responses prior to use.
Generated with Claude Code via /jira:solve [CNTRLPLANE-507](https://redhat.atlassian.net/browse/CNTRLPLANE-507)

---

Note: This PR was auto-generated by the jira-agent periodic CI job in response to CNTRLPLANE-507. See the full report for token usage, cost breakdown, and detailed phase output.

Summary by CodeRabbit

• Bug Fixes

  • Enhanced AWS PrivateLink cleanup coordination during HostedControlPlane deletion to ensure proper resource cleanup while credentials remain available.

• Tests

  • Added comprehensive test coverage for deletion handling and finalizer coordination.

[openshift-merge-bot]

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: LGTM mode

[openshift-ci-robot]

@hypershift-jira-solve-ci[bot]: This pull request references CNTRLPLANE-507 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "5.0.0" version, but no target version was set.

Details

In response to this:

What this PR does / why we need it:

Adds a finalizer on the HostedControlPlane resource from the AWSEndpointService reconciler to prevent HCP deletion before AWS PrivateLink resources are cleaned up.

Problem: When the CPO restarts during deletion of a SharedVPC cluster, the clientBuilder is uninitialized and the HCP (with its cross-account role ARNs) may already be deleted. This causes the reconciler to fail creating AWS clients, and after a 10-minute grace period the hypershift-operator force-removes the CPO finalizer — orphaning VPC endpoints, security groups, and DNS records in the shared VPC account.

Solution: The new HCP finalizer (hypershift.openshift.io/aws-private-link-endpoint-cleanup) follows the same pattern used by the Azure PLS controller:

• Adds the finalizer to the HCP during normal reconciliation
• When HCP deletion is detected, initializes AWS clients from the still-available HCP
• Cleans up each AWSEndpointService's AWS resources and removes CR finalizers
• Removes the HCP finalizer only after all AWSEndpointService CRs are cleaned up
• Extends the HCP watch handler (enqueueOnHCPChange) to also trigger reconciliation when an HCP is being deleted with the finalizer present

Which issue(s) this PR fixes:

Fixes https://redhat.atlassian.net/browse/CNTRLPLANE-507

Special notes for your reviewer:

• This follows the same finalizer pattern already established by the Azure PLS controller
• The enqueueOnHCPChange handler (renamed from enqueueOnAccessChange) now triggers on both EndpointAccess changes and HCP deletions with the finalizer
• AWS client initialization during HCP deletion reuses the existing getAWSClient helper, sourcing credentials from the still-available HCP spec

Checklist:

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Always review AI generated responses prior to use.
Generated with Claude Code via /jira:solve [CNTRLPLANE-507](https://redhat.atlassian.net/browse/CNTRLPLANE-507)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[coderabbitai]

📝 Walkthrough

Walkthrough

This change introduces an HCP-level finalizer (hcpAWSPrivateLinkFinalizerName) to coordinate AWS PrivateLink resource cleanup during HostedControlPlane deletion. The AWSEndpointService reconciler now watches both EndpointAccess changes and HCP deletion events. During reconciliation, an ensureHCPFinalizer helper adds the finalizer before normal processing. When HCP deletion is detected, a new reconcileHCPDeletion flow initializes AWS clients from HCP credentials, cleans up endpoint/security group/Route53 resources, and removes the HCP finalizer only after all AWSEndpointService CRs have cleared their finalizers. This ensures AWS credentials remain available throughout cleanup.

Sequence Diagram

sequenceDiagram
    participant K8s as Kubernetes API
    participant Rec as AWSEndpointService<br/>Reconciler
    participant HCP as HostedControlPlane
    participant AWS as AWS Services<br/>(EC2/Route53)

    rect rgba(100, 150, 200, 0.5)
    Note over Rec,HCP: Normal Reconciliation Path
    Rec->>K8s: Get HCP
    Rec->>Rec: ensureHCPFinalizer<br/>(add if missing)
    Rec->>K8s: Patch HCP with finalizer
    Rec->>Rec: Continue AWSEndpointService<br/>reconciliation
    end

    rect rgba(200, 100, 100, 0.5)
    Note over Rec,AWS: HCP Deletion Path
    K8s->>Rec: Detect HCP deletion<br/>(DeletionTimestamp set)
    Rec->>Rec: reconcileHCPDeletion
    Rec->>AWS: Initialize AWS client<br/>from HCP credentials
    Rec->>AWS: Delete endpoint service,<br/>security group, Route53 records
    Rec->>K8s: List all AWSEndpointService<br/>CRs in namespace
    alt All CR finalizers cleared
        Rec->>K8s: Remove HCP finalizer
        Rec->>K8s: HCP deletion proceeds
    else CR finalizers still present
        Rec->>Rec: Requeue reconciliation
    end
    end

Loading

Suggested reviewers

• jparrill
• jimdaga

🚥 Pre-merge checks | ✅ 11 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (11 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly identifies the main change: adding an HCP finalizer to the AWSEndpointService reconciler, which directly matches the core functionality introduced in the changeset.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Stable And Deterministic Test Names	✅ Passed	All newly added test case names use stable, deterministic strings with no dynamic values like timestamps, UUIDs, or indices. Test titles are descriptive and static as required.
Test Structure And Quality	✅ Passed	The custom check targets Ginkgo test code (Describe/It blocks, BeforeEach/AfterEach). The PR adds standard Go tests using testing.T and table-driven patterns, not Ginkgo. Check not applicable.
Microshift Test Compatibility	✅ Passed	The PR adds standard Go unit tests using *testing.T pattern with mocked clients, not Ginkgo e2e tests. The MicroShift Test Compatibility check targets only Ginkgo e2e tests, which are not present.
Single Node Openshift (Sno) Test Compatibility	✅ Passed	No Ginkgo e2e tests were added. The PR adds only standard Go unit tests. Custom check does not apply.
Topology-Aware Scheduling Compatibility	✅ Passed	PR modifies only controller reconciliation logic (finalizer/deletion handling). No deployment manifests, Pod specs, or scheduling constraints are introduced.
Ote Binary Stdout Contract	✅ Passed	Not applicable: PR modifies standard Kubernetes controller unit tests, not OTE binary code. No stdout violations detected.
Ipv6 And Disconnected Network Test Compatibility	✅ Passed	No Ginkgo e2e tests added. PR adds only standard Go unit tests using testing.T pattern. Custom check applies to Ginkgo e2e tests and is not applicable.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: hypershift-jira-solve-ci[bot]
Once this PR has been reviewed and has the lgtm label, please assign cblecker for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[codecov]

Codecov Report

❌ Patch coverage is 82.95455% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 40.05%. Comparing base (76bbefc) to head (80add24).

Files with missing lines	Patch %	Lines
...ollers/awsprivatelink/awsprivatelink_controller.go	82.95%	14 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #8499      +/-   ##
==========================================
+ Coverage   40.00%   40.05%   +0.05%     
==========================================
  Files         751      751              
  Lines       92838    92915      +77     
==========================================
+ Hits        37137    37216      +79     
+ Misses      53014    53009       -5     
- Partials     2687     2690       +3     

Files with missing lines	Coverage Δ
...ollers/awsprivatelink/awsprivatelink_controller.go	39.75% <82.95%> (+5.41%)	⬆️

Flag	Coverage Δ
cmd-support	34.09% <ø> (ø)
cpo-hostedcontrolplane	40.56% <ø> (ø)
cpo-other	40.53% <82.95%> (+0.39%)	⬆️
hypershift-operator	50.53% <ø> (ø)
other	31.54% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.