apiserver: sync with http server shutdown to flush existing connections

[sttts]

We did not wait for the http server to shut down completely. In the http/2 case this seems to lead to early closing of connections, without sending all bytes to the client. Credits to @xingxingxia for finding that.

Fix graceful apiserver shutdown to not drop outgoing bytes before the process terminates.

Here is the go doc of the http server shutdown method:

// Shutdown gracefully shuts down the server without interrupting any
// active connections. Shutdown works by first closing all open
// listeners, then closing all idle connections, and then waiting
// indefinitely for connections to return to idle and then shut down.
// If the provided context expires before the shutdown is complete,
// Shutdown returns the context's error, otherwise it returns any
// error returned from closing the Server's underlying Listener(s).
//
// When Shutdown is called, Serve, ListenAndServe, and
// ListenAndServeTLS immediately return ErrServerClosed. Make sure the
// program doesn't exit and waits instead for Shutdown to return.
//
// Shutdown does not attempt to close nor wait for hijacked
// connections such as WebSockets. The caller of Shutdown should
// separately notify such long-lived connections of shutdown and wait
// for them to close, if desired. See RegisterOnShutdown for a way to
// register shutdown notification functions.
//
// Once Shutdown has been called on a server, it may not be reused;
// future calls to methods such as Serve will return ErrServerClosed.
func (srv *Server) Shutdown(ctx context.Context) error {
...
}

[liggitt]

/lgtm
/approve

[k8s-ci-robot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: liggitt, sttts

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• cmd/cloud-controller-manager/OWNERS [liggitt,sttts]
• cmd/kube-controller-manager/OWNERS [liggitt,sttts]
• cmd/kube-scheduler/OWNERS [liggitt]
• staging/src/k8s.io/apiserver/OWNERS [liggitt,sttts]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[hzxuzhonghu]

/lgtm

[fedebongio]

/assign @logicalhan

[k8s-ci-robot]

New changes are detected. LGTM label has been removed.