kube-apiserver: use SO_REUSEPORT when creating listener

[invidian]

What type of PR is this?

Uncomment only one /kind <> line, hit enter to put that in a new line, and remove leading whitespace from that line:

/kind api-change
/kind bug
/kind cleanup
/kind deprecation
/kind design
/kind documentation
/kind failing-test

/kind feature

/kind flake

What this PR does / why we need it:

So multiple instances of kube-apiserver can bind on the same address and
port, to provide seamless upgrades.

Which issue(s) this PR fixes:

Fixes #88785

Special notes for your reviewer:

Perhaps some e2e tests could be added for that too, but I don't know how to do that.

Does this PR introduce a user-facing change?:

kube-apiserver, kube-scheduler and kube-controller manager now use SO_REUSEPORT socket option when listening on address defined by --bind-address and --secure-port flags, when running on Unix systems (Windows is NOT supported). This allows to run multiple instances of those processes on a single host with the same configuration, which allows to update/restart them in a graceful way, without causing downtime.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

[k8s-ci-robot]

Welcome @invidian!

It looks like this is your first PR to kubernetes/kubernetes 🎉. Please refer to our pull request process documentation to help your PR have a smooth ride to approval.

You will be prompted by a bot to use commands during the review process. Do not be afraid to follow the prompts! It is okay to experiment. Here is the bot commands documentation.

You can also check if kubernetes/kubernetes has its own contribution guidelines.

You may want to refer to our testing guide if you run into trouble with your tests not passing.

If you are having difficulty getting your pull request seen, please follow the recommended escalation practices. Also, for tips and tricks in the contribution process you may want to read the Kubernetes contributor cheat sheet. We want to make sure your contribution gets all the attention it needs!

Thank you, and welcome to Kubernetes. 😃

[k8s-ci-robot]

Hi @invidian. Thanks for your PR.

I'm waiting for a kubernetes member to verify that this patch is reasonable to test. If it is, they should reply with /ok-to-test on its own line. Until that is done, I will not automatically test new commits in this PR, but the usual testing commands by org members will still work. Regular contributors should join the org to skip this step.

Once the patch is verified, the new status will be reflected by the ok-to-test label.

I understand the commands that are listed here.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[sftim]

Does this change have an impact on portability?

/retest

[invidian]

Does this change have an impact on portability?

What kind of portability? OSes? Or CPU architectures? As far as I'm aware, this change will only work for Linux, I wasn't sure if I should target other OSes too (as mentioned in original issue).

It seems all tests fails with the same error. @sftim any hints how to reproduce it locally/why it fails?

[sftim]

@sftim any hints how to reproduce it locally/why it fails?

I'm afraid I don't have any insight into that.

[invidian]

I just run ./hack/update-bazel.sh, commited generated file and pushed again, maybe that's it? Can we re-test please?

[sftim]

/retest

[invidian]

I run now hack/update-vendor.sh and also pushed generated changes.

But I see now, that typecheck fails and I wonder if there is a way to disable it to ignore Windows or do I need to change the code to support windows as well 🤔

[invidian]

Ugh, fixed missing import in unit test (running tests locally now too, to make sure it's fine)

[invidian]

Ugh, fixed missing import in unit test (running tests locally now too, to make sure it's fine)

Also had to fix some other unit test types.

@lavalamp can you trigger tests again please?

[invidian]

Tests are passing now.

[lavalamp]

/approve
/lgtm

Thanks!

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: invidian, lavalamp

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/kube-scheduler/OWNERS [lavalamp]
• staging/src/k8s.io/apiserver/OWNERS [lavalamp]
• test/OWNERS [lavalamp]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[lavalamp]

(this won't merge until we exit code freeze btw)

[invidian]

Thanks!

Thanks for your guidance of my first K8s PR @lavalamp! I'm happy that it went in so smoothly.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[fejta-bot]

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel or /hold comment for consistent failures.

[mfojtik]

@invidian @lavalamp don't we also need SO_REUSEADDR to make "more than one instance to bind on the same address and port. [default=false]" true?

Also do we need some orchestration during upgrade to avoid old API server responding if it is not fully down (which can confuse health checks).

[invidian]

Also do we need some orchestration during upgrade to avoid old API server responding if it is not fully down (which can confuse health checks).

It has been discussed here: #88785 (comment). TL;DR it should be fine.

@invidian @lavalamp don't we also need SO_REUSEADDR to make "more than one instance to bind on the same address and port. [default=false]" true?

As far as I understand the difference between the two and from my testing, SO_REUSEPORT is sufficient to provide this functionality. Also as tests states.

[bhks]

Hi @invidian ,

Thank you for your PR.
I have question related to the health check of new API server coming up or the previous comment #88893 (comment).
Given you pointed out the answer to the comment #88785 (comment) but that comment answer mostly about the HA and version skew but not about the health checks.

When you get some time I am interested in learning about what happens if the new API server comes up but fails health due to issues.

My question is do we not need to first perform health check on new API server before start accepting actual traffic . Are we expecting the new binary to just work fine ?

[invidian]

My question is do we not need to first perform health check on new API server before start accepting actual traffic . Are we expecting the new binary to just work fine ?

It's a good question :) I don't know how this can be configured for health checks for individual instances. I usually rely on the fact, that if kube-apiserver is not functional, it restarts.

[bhks]

Thank you for your response!!!

Assuming its same IP and PORT on which API Server is being brought up:

So In cases of malfunction restart , when node have high traffic ~50% of the new connection can route to new API Server after it makes Listen() call without being accepted. Also we might not have mechanism to know if this new one is malfunctioning until we remove old running API Server.

If its different IP and same PORT then if it does not affect the traffic from my understanding.