New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

apiserver: sync with http server shutdown to flush existing connections #72970

Merged
merged 1 commit into from Jan 18, 2019

Conversation

@sttts
Copy link
Contributor

sttts commented Jan 16, 2019

We did not wait for the http server to shut down completely. In the http/2 case this seems to lead to early closing of connections, without sending all bytes to the client. Credits to @xingxingxia for finding that.

Fix graceful apiserver shutdown to not drop outgoing bytes before the process terminates.

Here is the go doc of the http server shutdown method:

// Shutdown gracefully shuts down the server without interrupting any
// active connections. Shutdown works by first closing all open
// listeners, then closing all idle connections, and then waiting
// indefinitely for connections to return to idle and then shut down.
// If the provided context expires before the shutdown is complete,
// Shutdown returns the context's error, otherwise it returns any
// error returned from closing the Server's underlying Listener(s).
//
// When Shutdown is called, Serve, ListenAndServe, and
// ListenAndServeTLS immediately return ErrServerClosed. Make sure the
// program doesn't exit and waits instead for Shutdown to return.
//
// Shutdown does not attempt to close nor wait for hijacked
// connections such as WebSockets. The caller of Shutdown should
// separately notify such long-lived connections of shutdown and wait
// for them to close, if desired. See RegisterOnShutdown for a way to
// register shutdown notification functions.
//
// Once Shutdown has been called on a server, it may not be reused;
// future calls to methods such as Serve will return ErrServerClosed.
func (srv *Server) Shutdown(ctx context.Context) error {
...
}

@sttts sttts force-pushed the sttts:sttts-wait-for-http-server-finished-shutdown branch from caca404 to 8ea8924 Jan 16, 2019

@liggitt

This comment has been minimized.

Copy link
Member

liggitt commented Jan 16, 2019

/lgtm
/approve

@k8s-ci-robot k8s-ci-robot added the lgtm label Jan 16, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 16, 2019

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@hzxuzhonghu

This comment has been minimized.

Copy link
Member

hzxuzhonghu commented Jan 17, 2019

/lgtm

@fedebongio

This comment has been minimized.

Copy link
Contributor

fedebongio commented Jan 17, 2019

/assign @logicalhan

@sttts sttts force-pushed the sttts:sttts-wait-for-http-server-finished-shutdown branch from 8ea8924 to 5b47f99 Jan 18, 2019

@k8s-ci-robot k8s-ci-robot removed the lgtm label Jan 18, 2019

@k8s-ci-robot

This comment has been minimized.

Copy link
Contributor

k8s-ci-robot commented Jan 18, 2019

New changes are detected. LGTM label has been removed.

@sttts sttts added the lgtm label Jan 18, 2019

@k8s-ci-robot k8s-ci-robot merged commit 26d58d3 into kubernetes:master Jan 18, 2019

18 checks passed

cla/linuxfoundation sttts authorized
Details
pull-kubernetes-bazel-build Job succeeded.
Details
pull-kubernetes-bazel-test Job succeeded.
Details
pull-kubernetes-cross Skipped
pull-kubernetes-e2e-gce Job succeeded.
Details
pull-kubernetes-e2e-gce-100-performance Job succeeded.
Details
pull-kubernetes-e2e-gce-device-plugin-gpu Job succeeded.
Details
pull-kubernetes-e2e-kops-aws Job succeeded.
Details
pull-kubernetes-e2e-kubeadm-gce Skipped
pull-kubernetes-godeps Job succeeded.
Details
pull-kubernetes-integration Job succeeded.
Details
pull-kubernetes-kubemark-e2e-gce-big Job succeeded.
Details
pull-kubernetes-local-e2e Skipped
pull-kubernetes-local-e2e-containerized Skipped
pull-kubernetes-node-e2e Job succeeded.
Details
pull-kubernetes-typecheck Job succeeded.
Details
pull-kubernetes-verify Job succeeded.
Details
tide In merge pool.
Details
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment