Add documentation for API server health checks #21185

johscheuer · 2020-05-25T13:19:04Z

This PR adds some documentation to the different "health" endpoints of the Kubernetes API server.

Initial implementation PR: kubernetes/kubernetes#78458

Fixes: #20888

netlify · 2020-05-25T13:29:10Z

Deploy preview for kubernetes-io-master-staging ready!

Built with commit f1588c0

https://deploy-preview-21185--kubernetes-io-master-staging.netlify.app

sftim · 2020-05-25T17:31:16Z

/sig api-machinery

sftim

Hi @johscheuer

Here's some feedback. The main thing you absolutely do need to change is a couple of hyperlinks, you should make these relative links.

I've also commented on whether or not these health check endpoints are aimed at human or machine clients. My guess, without looking at any k/kubernetes source code, is that the HTTP response code can be relied on but the rest of the response is not meant for machine parsing. It'd be good to clarify that.

As well as that there's some style feedback, most of which is there in case it's useful. I've highlighted all the feedback which is purely suggestions and that it's 100% OK for you to stick with your original wording.

I hope this is useful; please feel free to ask questions.

@logicalhan did you want to add anything here?

content/en/docs/reference/using-api/health-checks.md

johscheuer · 2020-05-27T05:25:16Z

Thanks for your time and your comments. I will go over them in the next days.

jimangel · 2020-06-10T15:27:09Z

@johscheuer do you still have time to update your PR based on Tim's review?

johscheuer · 2020-06-11T04:07:29Z

Thanks for the reminder. I will finish this today/tomorrow.

logicalhan · 2020-06-11T04:36:22Z

@logicalhan did you want to add anything here?

It looks like a reasonable starting point to me :)

I've also commented on whether or not these health check endpoints are aimed at human or machine clients. My guess, without looking at any k/kubernetes source code, is that the HTTP response code can be relied on but the rest of the response is not meant for machine parsing. It'd be good to clarify that.

Yeah that's about right. 200 means it's healthy/live/ready. This is going to be consumed mostly by the kubelet (or whatever thing you have that monitors your apiserver) which is going to decide based on the response code. The verbose flag is specifically intended to aid human debugging.

It is also probably worth documenting that individual health checks also expose http endpoints which are automatically exposed when the individual health check is registered to the corresponding health endpoint. In other words for each health check you see when you curl apiserver/livez:

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok

You are also provided apiserver/livez/<healthcheck-name> (which is the bit after the square brackets but before the ok). This should not be considered something you can depend on however and is exposed for human debugging. This is true for all health endpoints (i.e. livez,readyz,healthz).

For illustration, let's say apiserver is exposed on port 8080, then you can issue the following command:
curl localhost:8080/healthz/etcd

johscheuer · 2020-06-13T07:49:53Z

@sftim i reworked the site and integrated your feedback. Could you review it again?

sftim · 2020-06-13T18:50:58Z

I'm afraid we removed capture shortcodes.

@johscheuer can you rebase and revise to match?

sftim

Minor things

content/en/docs/reference/using-api/health-checks.md

sftim · 2020-06-15T21:42:00Z

/hold
following #21185 (comment)

This does need a rebase.

logicalhan · 2020-06-17T21:05:47Z

content/en/docs/reference/using-api/health-checks.md

+Each individual health check exposes an http endpoint and could can be checked individually.
+The schema for the individual health checks is `/livez/<healthcheck-name>` where `livez` and `readyz` and be used to indicate if you want to check thee liveness or the readiness of the API server.
+The `<healthcheck-name>` path can be discovered using the `verbose` flag from above and take the path between `[+]` and `ok`.
+These individual health checks should not be consumed by machines but can be helpful for a human operator to debug a system:


Can we explicitly call out that this is not a supported API (i.e. use these endpoints at your own discretion and we explicitly do not support forward API compatibility on these endpoints)?

Is there a feature state or similar to go with that @logicalhan? If it's viable to call them “alpha” then that's easy to do, the website even has a shortcode for that.

(If it's not tracked as a feature state, I feel that could be an issue against k/kubernetes so that at least the lack of tracking is visible).

it's not really a feature state.. these endpoints are actually pre-versioned, legacy endpoints which do not have the traditional API versioning guarantees.

Which is closer: alpha (don't rely on these, a future release could remove them without warning) or GA (you'll get plenty of official notice about deprecation)?

It sounds like the former. Are they perhaps insta-deprecated? I'd like to have some way of relating the stability to https://kubernetes.io/docs/reference/using-api/deprecation-policy/ - even if that means amending the deprecation policy page!

alpha seems reasonable.

sftim · 2020-06-21T13:57:02Z

Rebased
/hold cancel

sftim · 2020-06-21T13:58:38Z

@kubernetes/sig-api-machinery-pr-reviews do you have any more feedback on #21185 (comment) and the following comments?

johscheuer · 2020-06-29T04:57:32Z

Is there anything I can help with?

sftim · 2020-06-29T08:35:08Z

@johscheuer do you perhaps have a suggestion for someone who'd be a good choice for technical reviewer? If so, we can ask them to take a look.

sttts · 2020-06-29T08:58:51Z

do you perhaps have a suggestion for someone who'd be a good choice for technical reviewer?

@logicalhan has certainly most insight into this.

sftim · 2020-06-29T17:31:38Z

@johscheuer please revise in line with #21185 (comment)

zparnold · 2020-07-01T01:01:55Z

@johscheuer when you get a chance can you please look at this? ☝️

johscheuer · 2020-07-06T10:53:19Z

I make the requested changes in the next few days!

johscheuer · 2020-07-21T08:25:04Z

Sorry for the delay I added the feature state information.

jimangel · 2020-07-27T03:03:21Z

/lgtm
/approve

k8s-ci-robot · 2020-07-27T03:03:35Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: jimangel

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~content/en/OWNERS~~ [jimangel]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

k8s-ci-robot added the cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. label May 25, 2020

k8s-ci-robot requested review from makoscafee and Rajakavitha1 May 25, 2020 13:19

k8s-ci-robot added language/en Issues or PRs related to English language size/M Denotes a PR that changes 30-99 lines, ignoring generated files. sig/docs Categorizes an issue or PR as relevant to SIG Docs. labels May 25, 2020

johscheuer mentioned this pull request May 25, 2020

Update kube-apiserver flag comments kubernetes/kubernetes#91409

Merged

k8s-ci-robot added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label May 25, 2020

sftim reviewed May 25, 2020

View reviewed changes

johscheuer force-pushed the add-docs-api-health-checks branch from becea86 to 500e297 Compare June 13, 2020 07:48

sftim reviewed Jun 13, 2020

View reviewed changes

content/en/docs/reference/using-api/health-checks.md Show resolved Hide resolved

content/en/docs/reference/using-api/health-checks.md Show resolved Hide resolved

k8s-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 15, 2020

johscheuer force-pushed the add-docs-api-health-checks branch from 500e297 to 314ad55 Compare June 16, 2020 09:18

Add documentation for API server health checks

8308aab

johscheuer force-pushed the add-docs-api-health-checks branch from 314ad55 to 8308aab Compare June 16, 2020 13:47

logicalhan reviewed Jun 17, 2020

View reviewed changes

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jun 21, 2020

Add feature state for individual health checks

f1588c0

k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Jul 21, 2020

k8s-ci-robot assigned jimangel Jul 27, 2020

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Jul 27, 2020

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jul 27, 2020

k8s-ci-robot merged commit 5c27482 into kubernetes:master Jul 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation for API server health checks #21185

Add documentation for API server health checks #21185

johscheuer commented May 25, 2020

netlify bot commented May 25, 2020 •

edited

sftim commented May 25, 2020

sftim left a comment

johscheuer commented May 27, 2020

jimangel commented Jun 10, 2020

johscheuer commented Jun 11, 2020

logicalhan commented Jun 11, 2020 •

edited

johscheuer commented Jun 13, 2020

sftim commented Jun 13, 2020

sftim left a comment

sftim commented Jun 15, 2020

logicalhan Jun 17, 2020

sftim Jun 17, 2020

logicalhan Jun 29, 2020

sftim Jun 29, 2020

logicalhan Jun 29, 2020

sftim commented Jun 21, 2020

sftim commented Jun 21, 2020

johscheuer commented Jun 29, 2020

sftim commented Jun 29, 2020

sttts commented Jun 29, 2020

sftim commented Jun 29, 2020

zparnold commented Jul 1, 2020

johscheuer commented Jul 6, 2020

johscheuer commented Jul 21, 2020

jimangel commented Jul 27, 2020

k8s-ci-robot commented Jul 27, 2020

Add documentation for API server health checks #21185

Add documentation for API server health checks #21185

Conversation

johscheuer commented May 25, 2020

netlify bot commented May 25, 2020 • edited

sftim commented May 25, 2020

sftim left a comment

Choose a reason for hiding this comment

johscheuer commented May 27, 2020

jimangel commented Jun 10, 2020

johscheuer commented Jun 11, 2020

logicalhan commented Jun 11, 2020 • edited

johscheuer commented Jun 13, 2020

sftim commented Jun 13, 2020

sftim left a comment

Choose a reason for hiding this comment

sftim commented Jun 15, 2020

logicalhan Jun 17, 2020

Choose a reason for hiding this comment

sftim Jun 17, 2020

Choose a reason for hiding this comment

logicalhan Jun 29, 2020

Choose a reason for hiding this comment

sftim Jun 29, 2020

Choose a reason for hiding this comment

logicalhan Jun 29, 2020

Choose a reason for hiding this comment

sftim commented Jun 21, 2020

sftim commented Jun 21, 2020

johscheuer commented Jun 29, 2020

sftim commented Jun 29, 2020

sttts commented Jun 29, 2020

sftim commented Jun 29, 2020

zparnold commented Jul 1, 2020

johscheuer commented Jul 6, 2020

johscheuer commented Jul 21, 2020

jimangel commented Jul 27, 2020

k8s-ci-robot commented Jul 27, 2020

netlify bot commented May 25, 2020 •

edited

logicalhan commented Jun 11, 2020 •

edited