Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

kube-apiserver: don't create endpoints before being ready #74668

Merged
merged 2 commits into from Mar 4, 2019

Conversation

@sttts
Copy link
Contributor

sttts commented Feb 27, 2019

We had a race that the kube-apiserver was creating its endpoints before being ready to serve requests. This PR adds endpoint removal very early in the startup in order to remove bad endpoints after a crash. Then it postpones the recreation of the endpoints until the server is actually reporting ready via /healthz.

Fix kube-apiserver not to create default/kubernetes service endpoints before it reports readiness via the /healthz and therefore is ready to serve requests. Also early during startup old endpoints are remove which might be left over from a previously crashed kube-apiserver.
Show resolved Hide resolved pkg/master/controller.go Outdated

@sttts sttts force-pushed the sttts:sttts-kube-apiserver-endpoints-when-ready branch from 342312f to 748007a Feb 27, 2019

Show resolved Hide resolved pkg/master/controller.go Outdated
@deads2k

This comment has been minimized.

Copy link
Contributor

deads2k commented Feb 27, 2019

This makes us consistent with how normal services work. @liggitt may want some more specific text in that release-note since the behavior does change.

@kubernetes/sig-api-machinery-bugs

lgtm, but I'll give extra time for reviewers to look at the behavior change. @lavalamp @logicalhan

@k8s-ci-robot k8s-ci-robot added kind/bug and removed needs-kind labels Feb 27, 2019

@sttts sttts force-pushed the sttts:sttts-kube-apiserver-endpoints-when-ready branch from 748007a to 8d9d138 Feb 27, 2019

@sttts sttts added this to the v1.14 milestone Feb 27, 2019

@lavalamp

This comment has been minimized.

Copy link
Member

lavalamp commented Feb 27, 2019

Ugh, it's so criminal that we manage endpoints specially for apiserver.

/assign @logicalhan

@sttts sttts force-pushed the sttts:sttts-kube-apiserver-endpoints-when-ready branch from 8d9d138 to a9dc5fd Feb 27, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Feb 27, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@sttts

This comment has been minimized.

Copy link
Contributor Author

sttts commented Feb 27, 2019

/retest

@sttts

This comment has been minimized.

Copy link
Contributor Author

sttts commented Feb 28, 2019

@logicalhan lgty?

@@ -138,6 +142,12 @@ func (c *Controller) Start() {
return
}

// Reconcile during first run removing itself until server is ready.
endpointPorts := createEndpointPortSpec(c.PublicServicePort, "https", c.ExtraEndpointPorts)
if err := c.EndpointReconciler.RemoveEndpoints(kubernetesServiceName, c.PublicIP, endpointPorts); err != nil {

This comment has been minimized.

@logicalhan

logicalhan Feb 28, 2019

Contributor

Should this be a failure condition of the poststarthook? What is the expected behavior if we fail to remove an endpoint properly during the boot cycle?

This comment has been minimized.

@sttts

sttts Mar 1, 2019

Author Contributor

it wasn't before, compare 15 lines down.

This comment has been minimized.

@sttts

sttts Mar 1, 2019

Author Contributor

also this line is best effort. If it fails (e.g. due to etcd connection errors), it has no consequences. And if it fails, we probably have much worse problems than dangling endpoints. I would leave it with an error messages.

If we need more advanced logic, let's move that to a follow-up. I would perfer to keep this PR contained.

Show resolved Hide resolved pkg/master/reconcilers/reconcilers.go Outdated
@roycaihw

This comment has been minimized.

Copy link
Member

roycaihw commented Feb 28, 2019

@sttts sttts force-pushed the sttts:sttts-kube-apiserver-endpoints-when-ready branch from a9dc5fd to 2a9a9fa Mar 1, 2019

@mfojtik

This comment has been minimized.

Copy link
Contributor

mfojtik commented Mar 4, 2019

/lgtm

@k8s-ci-robot k8s-ci-robot added the lgtm label Mar 4, 2019

@k8s-ci-robot k8s-ci-robot merged commit b1d4d40 into kubernetes:master Mar 4, 2019

16 checks passed

cla/linuxfoundation sttts authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped.
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-godeps Skipped.
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped.
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
pull-publishing-bot-validate Skipped.
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
You can’t perform that action at this time.