OCPBUGS-33428: Reconcile KAS endpoints and endpoint slice

[rtheis]

What this PR does / why we need it:

See issue for details and test results.

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes # https://issues.redhat.com/browse/OCPBUGS-33428

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

[stevekuznetsov]

Code seems straightforward - is this associated with a Jira ticket?

[rtheis]

Code seems straightforward - is this associated with a Jira ticket?

@stevekuznetsov Thanks for the review. I will address the comments and will be opening a Jira with the details. I wanted to pull together some test results with background information as to why I think we need this change.

[rtheis]

/retitle OCPBUGS-33428: Reconcile KAS endpoints and endpoint slice

[openshift-ci-robot]

@rtheis: This pull request references Jira Issue OCPBUGS-33428, which is invalid:

• expected the bug to target the "4.16.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

What this PR does / why we need it:

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes #

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[netlify]

✅ Deploy Preview for hypershift-docs ready!

Name	Link
🔨 Latest commit	225b286
🔍 Latest deploy log	https://app.netlify.com/sites/hypershift-docs/deploys/663ca680dd6caa000817e273
😎 Deploy Preview	https://deploy-preview-3942--hypershift-docs.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site configuration.

[rtheis]

/test e2e-aws

[rtheis]

/test e2e-azure

[jeffnowicki]

/jira refresh

[openshift-ci-robot]

@jeffnowicki: This pull request references Jira Issue OCPBUGS-33428, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug

• bug is open, matching expected state (open)
• bug target version (4.16.0) matches configured target version for branch (4.16.0)
• bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[stevekuznetsov]

I really wish we had some way to test the whole reconciler together so we could capture that it now does this. @sjenning

[stevekuznetsov]

/lgtm

[enxebre]

Why is this issue only hit when having haproxy to intercept traffic to the endpoint? why is it not hit with a traditional svc->endpoint->pod flow?
Is it because we set advertise-address explicitly and that results in a single address for the endpoint hence the issue in the ticket when one single instance goes down, or is it something else?

I'm concern we opt out to other goodies by setting none for --endpoint-reconciler-type

Other than the concerns above, code changes lgtm

[rtheis]

Why is this issue only hit when having haproxy to intercept traffic to the endpoint? why is it not hit with a traditional svc->endpoint->pod flow? Is it because we set advertise-address explicitly and that results in a single address for the endpoint hence the issue in the ticket when one single instance goes down, or is it something else?

I'm concern we opt out to other goodies by setting none for --endpoint-reconciler-type

Other than the concerns above, code changes lgtm

@enxebre My understanding is that setting the KAS advertise-address results in a single address for the KAS endpoint in the data plane regardless of there being multiple KAS pods/endpoints in the control plane. As a result, when any KAS pod is terminated, the pod will now remove its address from the endpoint. In previous releases, the KAS would see a single address and refuse to remove its address from the endpoint. That failsafe was removed by Kubernetes and thus we see single endpoint setups, such as used by HyperShift, encounter this KAS in cluster availability problem during routine maintenance activities. Based on our discussions with the Kubernetes community via kubernetes/kubernetes#118777, I think we unfortunately need to set none for --endpoint-reconciler-type and manage the KAS endpoints and endpoint slice ourselves.

[enxebre]

/approve

[openshift-ci]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: enxebre, rtheis, stevekuznetsov

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• OWNERS [enxebre]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[stevekuznetsov]

/lgtm

[openshift-ci-robot]

/retest-required

Remaining retests: 0 against base HEAD 239a333 and 2 for PR HEAD 225b286 in total

[rtheis]

/retest-required

[rtheis]

/retest-required

[rtheis]

/retest-required

[rtheis]

/retest-required

[rtheis]

/retest-required

[rtheis]

/retest-required

[openshift-ci]

@rtheis: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci-robot]

@rtheis: Jira Issue OCPBUGS-33428: All pull requests linked via external trackers have merged:

• openshift/hypershift#3942

Jira Issue OCPBUGS-33428 has been moved to the MODIFIED state.

In response to this:

What this PR does / why we need it:

See issue for details and test results.

Which issue(s) this PR fixes (optional, use fixes #<issue_number>(, fixes #<issue_number>, ...) format, where issue_number might be a GitHub issue, or a Jira story:
Fixes # https://issues.redhat.com/browse/OCPBUGS-33428

Checklist

• Subject and description added to both, commit and PR.
• Relevant issues have been referenced.
• This change includes docs.
• This change includes unit tests.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[rtheis]

/cherry-pick release-4.16

[rtheis]

/cherry-pick release-4.15

[openshift-cherrypick-robot]

@rtheis: new pull request created: #4096

In response to this:

/cherry-pick release-4.16

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-cherrypick-robot]

@rtheis: new pull request created: #4097

In response to this:

/cherry-pick release-4.15

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[openshift-bot]

[ART PR BUILD NOTIFIER]

This PR has been included in build ose-hypershift-container-v4.17.0-202405271841.p0.g4c2f51c.assembly.stream.el9 for distgit hypershift.
All builds following this will include this PR.