Implement a new shared "Drainer" handler. #1517
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
knative-prow-robot
added
the
size/L
knative-prow-robot
added
the
approved
vagababov
reviewed
Jul 17, 2020
Comment on lines
+38
to
+43
| Inner http.Handler | ||
|
|
||
| // QuietPeriod is the duration that must elapse without any requests | ||
| // after Drain is called before it may return. | ||
| QuietPeriod time.Duration | ||
|
|
There was a problem hiding this comment.
I'd hide those as well and add a ctor
There was a problem hiding this comment.
Makes it more complicated to have an optional QuietPeriod 🤷
This implements a new `http.Handler` called `Drainer`, which is intended to wrap some inner `http.Handler` business logic with a new outer handler that can respond to Kubelet probes (successfully until told to "Drain()"). This takes over the webhook's relatively new probe handling and lame duck logic with one key difference. Previously the webhook waited for a fixed period after SIGTERM before exitting, but the new logic waits for this same grace period AFTER THE LAST REQUEST. So if the handler keeps getting (non-probe) requests, the timer will continually reset, and once it stops receiving requests for the configured grace period, "Drain()" will return and the webhook will exit. The goal of this work is to try to better cope with what we believe to be high tail latencies of the API server seeing that a webhook replica is shutting down. Related: knative#1509
|
The following is the coverage report on the affected files.
|
vagababov
reviewed
Jul 18, 2020
Comment on lines
+38
to
+43
| Inner http.Handler | ||
|
|
||
| // QuietPeriod is the duration that must elapse without any requests | ||
| // after Drain is called before it may return. | ||
| QuietPeriod time.Duration | ||
|
|
knative-prow-robot
added
do-not-merge/hold
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: mattmoor, vagababov The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
|
/unhold |
knative-prow-robot
removed
the
do-not-merge/hold
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This implements a new
http.HandlercalledDrainer, which is intended to wrap some innerhttp.Handlerbusiness logic with a new outer handler that can respond to Kubelet probes (successfully until told to "Drain()").This takes over the webhook's relatively new probe handling and lame duck logic with one key difference. Previously the webhook waited for a fixed period after SIGTERM before exitting, but the new logic waits for this same grace period AFTER THE LAST REQUEST. So if the handler keeps getting (non-probe) requests, the timer will continually reset, and once it stops receiving requests for the configured grace period, "Drain()" will return and the webhook will exit.
The goal of this work is to try to better cope with what we believe to be high tail latencies of the API server seeing that a webhook replica is shutting down.
Related: #1509