load balancer health check for kube-apiserver #3537

tkashem · 2020-05-03T22:00:38Z

No description provided.

tkashem · 2020-05-04T14:13:15Z

@abhinavdahiya let me know if this is the right place for this doc, otherwise I will move it.

abhinavdahiya · 2020-05-04T22:19:22Z

Thanks for the detailed doc @tkashem !

I think the next step will be to make this doc discoverable by linking this doc from the code that defines these healthchecks in data/data/{aws,gcp} ..

docs/dev/kube-apiserver-health-check.md

sttts · 2020-05-05T07:22:33Z

docs/dev/kube-apiserver-health-check.md

+- Existing connections are not cut off hard, they are allowed to complete gracefully.
+
+## Load Balancer Health Check Probe
+`kube-apiserver` provides graceful termination support via the `/readyz` health check endpoint. When `/readyz`reports


nit: space in front of "reports"

sttts · 2020-05-05T07:23:34Z

docs/dev/kube-apiserver-health-check.md

+
+Now let's walk through the events (in chronological order) that unfold when a `kube-apiserver` instance restarts:
+* E1: `T+0s`: `kube-apiserver` receives a TERM signal.
+* E2: `T+0s`: `/readyz` starts reporting `failure` to signal to the load balancer that a shut down is in progress.


everywhere in the doc: load balancers

sttts · 2020-05-05T07:24:50Z

docs/dev/kube-apiserver-health-check.md

+* E3: `T+70s`: `kube-apiserver` (the http server) stops listening:
+  * `/healthz` turns red.
+  * Default TCP health check probe on port `6443` will fail.
+  * Any new request forwarded to it will fail, most likely with a `connection refused` error.


or GOAWAY for http/2

docs/dev/kube-apiserver-health-check.md

sttts · 2020-05-05T07:26:10Z

docs/dev/kube-apiserver-health-check.md

+* E5: `T+70s+60s`: The apiserver process exits.
+
+An important note to consider is that today the time difference between `E3` and `E2` is `70s`. This is known as
+`shutdown-delay-duration` and is configurable.


be clear: not configurable by the user, but by the devs

sttts · 2020-05-05T07:28:32Z

docs/dev/kube-apiserver-health-check.md

+Let's plot the events as it happens when a load balance determines that a `kube-apiserver` is unhealthy and 
+takes it out of service. This considers a worst case scenario:
+* P1: T+0s:  `/readyz` starts reporting red.
+* P2: T+10s: first probe initiated.


be clear that this is an example. P2 could be right at T+0s, depending on the alignment of the probe request interval.

made it clear that this is a worst case scenario to calculate at most 30s

sttts · 2020-05-05T07:29:37Z

docs/dev/kube-apiserver-health-check.md

+
+We assume the following:
+* Each health check request is independent and lasts the entire interval.
+* The time it takes for the instance to respond does not affect the interval for the next health check.


do we know from aws/gcp docs that this is really the case?

this is true for aws, i copied them verbatim from aws doc.

link the docs, to make this easy to check? Seems like it's the classic-LB docs, but the installer uses network load balancers (classic LBs are aws_elb).

I could not find a doc that describes this for network load balancer exclusively. I think the health check mechanics should be the same for classic, application and network LB. Maybe we can ask this question to our AWS account rep.

On the other hand, what we stipulate above must hold true for all health checks universally. Otherwise if we allow one interval to bleed into another then we don't have a deterministic "at most".

wking · 2020-05-07T19:31:23Z

docs/dev/kube-apiserver-health-check.md

+
+## Load Balancer Health Check Probe
+`kube-apiserver` provides graceful termination support via the `/readyz` health check endpoint. When `/readyz` reports
+`ok` it indicates that the apiserver is ready to serve request(s).


nit: ok -> 200 OK? I expect LBs to care about HTTP status codes and not about the response body. And your ok is likely shorthand for the 200 status, but I think explicitly saying "200" (and possibly even "HTTP status 200 OK") would make it harder to misunderstand.

Thanks for the detailed doc @tkashem !
I think the next step will be to make this doc discoverable by linking this doc from the code that defines these healthchecks in data/data/{aws,gcp} ..

@abhinavdahiya I linked the doc.

wking · 2020-05-07T19:34:22Z

docs/dev/kube-apiserver-health-check.md

+* E2: `T+0s`: `/readyz` starts reporting `failure` to signal to the load balancers that a shut down is in progress.
+  * The apiserver will continue to accept new request(s).
+  * The apiserver waits for certain amount of time (configurable by `shutdown-delay-duration`) before it stops accepting new request(s).
+* E3: `T+70s`: `kube-apiserver` (the http server) stops listening:


Elsewhere in the doc you have:

In future we will reduce shutdown-delay-duration to 30s.

I'd rather make this portion of the doc robust to that sort of pivot by using T+shudown-delay-duration here.

As far as the user/dev is concerned, they should treat shutdown-delay-duration to be 30s for the purpose of designing health check probes. So I changed it to T+30s.

wking · 2020-05-07T19:35:52Z

docs/dev/kube-apiserver-health-check.md

+  * `/healthz` turns red.
+  * Default TCP health check probe on port `6443` will fail.
+  * Any new request forwarded to it will fail, most likely with a `connection refused` error or `GOAWAY` for http/2.
+  * Existing request(s) in-flight are not cut off but are given up to `60s` to complete gracefully.


Does this 60s also have a config variable name that we can use to guard against future default changes?

This is hardcoded in kube-apiserver.

sttts · 2020-05-07T19:39:54Z

lgtm

abhinavdahiya · 2020-05-08T23:37:48Z

Thanks for the detailed doc @tkashem !

I think the next step will be to make this doc discoverable by linking this doc from the code that defines these healthchecks in data/data/{aws,gcp} ..

@tkashem hopefully you saw this.

Per [0], the /readyz endpoint is how the api communicates that it is gracefully shutting down. Once /readyz starts to report failure, we want to stop sending traffic to that backend. If we wait for /healthz, it may be too late because once /healthz starts failing the api is already not accepting connections. 0: openshift/installer#3537

Per [0], the /readyz endpoint is how the api communicates that it is gracefully shutting down. Once /readyz starts to report failure, we want to stop sending traffic to that backend. If we wait for /healthz, it may be too late because once /healthz starts failing the api is already not accepting connections. I also moved the liveness probe for haproxy itself to use a /readyz endpoint for consistency. This isn't strictly necessary, but I think it will be less confusing if there aren't multiple health check endpoints in the config. 0: openshift/installer#3537

abhinavdahiya · 2020-05-11T20:14:39Z

upi/gcp/02_lb_ext.py

@@ -1,5 +1,6 @@
 def GenerateConfig(context):

+	// Refer to docs/dev/kube-apiserver-health-check.md on how to correctly setup health check probe for kube-apiserver


this is probably not correct comment syntax in python

oops, my bad. fixed.

abhinavdahiya · 2020-05-11T20:14:50Z

upi/gcp/02_lb_int.py

@@ -6,6 +6,7 @@ def GenerateConfig(context):
            'group': '$(ref.' + context.properties['infra_id'] + '-master-' + zone + '-instance-group' + '.selfLink)'
        })

+	// Refer to docs/dev/kube-apiserver-health-check.md on how to correctly setup health check probe for kube-apiserver


same as https://github.com/openshift/installer/pull/3537/files#r423292196

abhinavdahiya · 2020-05-11T20:15:12Z

/test e2e-gcp-upi

openshift-ci-robot · 2020-05-11T21:25:53Z

@tkashem: The following test failed, say /retest to rerun all failed tests:

Test name	Commit	Details	Rerun command
ci/prow/e2e-gcp-upi	`49cb2af`	link	`/test e2e-gcp-upi`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

abhinavdahiya · 2020-05-12T13:52:29Z

/approve
/lgtm

abhinavdahiya · 2020-05-12T13:52:59Z

Adding valid bug since this is docs update

openshift-ci-robot · 2020-05-12T13:54:00Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: abhinavdahiya

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [abhinavdahiya]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

openshift-bot · 2020-05-12T14:01:39Z

/retest

Please review the full test history for this PR and help us cut down flakes.

Per [0], the /readyz endpoint is how the api communicates that it is gracefully shutting down. Once /readyz starts to report failure, we want to stop sending traffic to that backend. If we wait for /healthz, it may be too late because once /healthz starts failing the api is already not accepting connections. I also moved the liveness probe for haproxy itself to use a /readyz endpoint for consistency. This isn't strictly necessary, but I think it will be less confusing if there aren't multiple health check endpoints in the config. 0: openshift/installer#3537 (cherry picked from commit 022933c)

openshift-ci-robot requested review from jhixson74 and mtnbikenc May 3, 2020 22:00

openshift-ci-robot assigned abhinavdahiya May 4, 2020

abhinavdahiya requested review from abhinavdahiya and removed request for jhixson74 and mtnbikenc May 4, 2020 22:19

abhinavdahiya closed this May 4, 2020

abhinavdahiya reopened this May 4, 2020

sttts reviewed May 5, 2020

View reviewed changes

docs/dev/kube-apiserver-health-check.md Show resolved Hide resolved

sttts reviewed May 5, 2020

View reviewed changes

docs/dev/kube-apiserver-health-check.md Outdated Show resolved Hide resolved

sttts reviewed May 5, 2020

View reviewed changes

tkashem force-pushed the kube-apiserver-health-check branch 4 times, most recently from 28371c9 to b8d4bb5 Compare May 6, 2020 18:50

wking reviewed May 7, 2020

View reviewed changes

cybertron mentioned this pull request May 11, 2020

Bug 1823950: Switch to /readyz for haproxy healthchecking openshift/baremetal-runtimecfg#62

Closed

cybertron mentioned this pull request May 11, 2020

Bug 1823950: [baremetal] Switch to /readyz for haproxy healthchecking openshift/machine-config-operator#1724

Merged

load balancer health check for kube-apiserver

dcd415c

tkashem force-pushed the kube-apiserver-health-check branch from b8d4bb5 to dcd415c Compare May 11, 2020 19:36

abhinavdahiya reviewed May 11, 2020

View reviewed changes

Make the doc discoverable from where health checks are defined

3bc71bb

tkashem force-pushed the kube-apiserver-health-check branch from 49cb2af to 3bc71bb Compare May 11, 2020 20:29

openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label May 12, 2020

openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label May 12, 2020

abhinavdahiya added bugzilla/valid-bug Indicates that a referenced Bugzilla bug is valid for the branch this PR is targeting. retest-not-required-docs-only labels May 12, 2020

openshift-merge-robot merged commit a1a6300 into openshift:master May 12, 2020

abhinavdahiya mentioned this pull request May 13, 2020

gcp: document gcp-routes service interdependency #3512

Closed

EgorLu mentioned this pull request Aug 10, 2020

Bug 1840366: [baremetal] Switch to /readyz for haproxy healthchecking openshift/machine-config-operator#1997

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load balancer health check for kube-apiserver #3537

load balancer health check for kube-apiserver #3537

tkashem commented May 3, 2020

tkashem commented May 4, 2020

abhinavdahiya commented May 4, 2020

sttts May 5, 2020

tkashem May 6, 2020

sttts May 5, 2020

tkashem May 6, 2020

sttts May 5, 2020

tkashem May 6, 2020

sttts May 5, 2020

tkashem May 6, 2020

sttts May 5, 2020

tkashem May 6, 2020

sttts May 5, 2020

tkashem May 6, 2020

wking May 7, 2020

tkashem May 11, 2020

wking May 7, 2020 •

edited

tkashem May 11, 2020

wking May 7, 2020

tkashem May 11, 2020

wking May 7, 2020

sttts May 7, 2020

sttts commented May 7, 2020

abhinavdahiya commented May 8, 2020

abhinavdahiya May 11, 2020

tkashem May 11, 2020

abhinavdahiya May 11, 2020

tkashem May 11, 2020

abhinavdahiya commented May 11, 2020

openshift-ci-robot commented May 11, 2020 •

edited

abhinavdahiya commented May 12, 2020

abhinavdahiya commented May 12, 2020

openshift-ci-robot commented May 12, 2020

openshift-bot commented May 12, 2020

		@@ -1,5 +1,6 @@
		def GenerateConfig(context):

		// Refer to docs/dev/kube-apiserver-health-check.md on how to correctly setup health check probe for kube-apiserver

load balancer health check for kube-apiserver #3537

load balancer health check for kube-apiserver #3537

Conversation

tkashem commented May 3, 2020

tkashem commented May 4, 2020

abhinavdahiya commented May 4, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

wking May 7, 2020 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sttts commented May 7, 2020

abhinavdahiya commented May 8, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

abhinavdahiya commented May 11, 2020

openshift-ci-robot commented May 11, 2020 • edited

abhinavdahiya commented May 12, 2020

abhinavdahiya commented May 12, 2020

openshift-ci-robot commented May 12, 2020

openshift-bot commented May 12, 2020

wking May 7, 2020 •

edited

openshift-ci-robot commented May 11, 2020 •

edited