Expose kubelet health checks using new prometheus endpoint #61369

rramkumar1 · 2018-03-19T21:48:52Z

What this PR does / why we need it:
Expose the results of kubelet liveness and readiness probes through a new endpoint on the kubelet called /containerHealth. This endpoint will expose a Prometheus metric. Below is a snippet of output when that endpoint is queried.

rramkumar@e2e-test-rramkumar-master ~ $ curl localhost:10255/metrics/probes
# HELP prober_probe_result The result of a liveness or readiness probe for a container.
# TYPE prober_probe_result gauge
prober_probe_result{container_name="kube-apiserver",namespace="kube-system",pod_name="kube-apiserver-e2e-test-rramkumar-master",pod_uid="949e11ad296ad9e3c842fd900f8cc723",probe_type="Liveness"} 0
prober_probe_result{container_name="kube-controller-manager",namespace="kube-system",pod_name="kube-controller-manager-e2e-test-rramkumar-master",pod_uid="0abfc37840bba279706ec39ae53a924c",probe_type="Liveness"} 0
prober_probe_result{container_name="kube-scheduler",namespace="kube-system",pod_name="kube-scheduler-e2e-test-rramkumar-master",pod_uid="0cd4171f9c806808291e6e24f99f0454",probe_type="Liveness"} 0
prober_probe_result{container_name="l7-lb-controller",namespace="kube-system",pod_name="l7-lb-controller-v0.9.8-alpha.2-e2e-test-rramkumar-master",pod_uid="968c792f4c1772566c71403dca2407f9",probe_type="Liveness"} 0

Which issue(s) this PR fixes (optional, in fixes #<issue number>(, fixes #<issue_number>, ...) format, will close the issue(s) when PR gets merged):
Fixes #58235

Release note:

Kubelet now exposes a new endpoint /metrics/probes which exposes a Prometheus metric containing the liveness and/or readiness probe results for a container.

rramkumar1 · 2018-03-19T21:49:53Z

/assign @loburm
For initial review of instrumentation piece.

loburm

Overall looks OK for me.

loburm · 2018-03-21T10:46:23Z

pkg/kubelet/prober/results/results_manager.go

@@ -58,6 +58,17 @@ func (r Result) String() string {
 	}
 }

+func (r Result) ToPrometheusType() float64 {


Please add a comment to this function.

loburm · 2018-03-21T11:08:27Z

pkg/kubelet/prober/prober_manager.go

+		Name:      "probe_result",
+		Help:      "The result of a liveness or readiness probe for a container.",
+	},
+	[]string{"probe_type", "container_name", "pod_name", "namespace", "pod_uid"},


not sure if we need both pod_uid and namespace+pod_name each of this should uniquely identify pod.

I think in #58827 we decided that it would be good to have both?

rramkumar1 · 2018-03-21T15:37:25Z

/assign @tallclair

tallclair · 2018-03-26T21:13:41Z

pkg/kubelet/server/server.go

@@ -66,6 +67,7 @@ import (
 const (
 	metricsPath         = "/metrics"
 	cadvisorMetricsPath = "/metrics/cadvisor"
+	containerHealthPath = "/containerHealth"


Why is this exposed on a new endpoint, rather than /metrics? Minimally I think this should be /metrics/probes, but maybe it can just be in metrics?

@loburm did not want this under /metrics. His reasoning was that anything under that endpoint should be metrics about the operation of kubelet only, not about its probes. Maybe we should revisit this discussion?

This question was discussed on sig-instrumentation meeting and we have agreed that /metrics endpoint should not expose information about state of containers running on the node. This endpoint contains only kubelet operational metrics and we wanted to keep it like that.

Ack. What do you think about /metrics/probes then? /metrics/cadvisor already has per-container metrics, I believe.

I'm fine with /metrics/probes. @loburm does that sound okay?

tallclair

one minor nit, then LGTM

tallclair · 2018-03-30T18:52:48Z

pkg/kubelet/prober/prober_manager.go

@@ -31,6 +32,16 @@ import (
 	"k8s.io/kubernetes/pkg/kubelet/util/format"
 )

+// ProberResults stores the results of a probe as prometheus metrics.
+var ProberResults = prometheus.NewGaugeVec(
+	prometheus.GaugeOpts{


Should this set a namespace?

I don't think any of the metrics declared in pkg/kubelet set the namespace so to be consistent I don't think I should here.

tallclair · 2018-03-31T01:41:59Z

/lgtm

k8s-ci-robot · 2018-03-31T01:42:05Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: rramkumar1, tallclair

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~pkg/kubelet/OWNERS~~ [tallclair]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

fejta-bot · 2018-03-31T03:56:50Z

/retest
This bot automatically retries jobs that failed/flaked on approved PRs (send feedback to fejta).

Review the full test history for this PR.

Silence the bot with an /lgtm cancel comment for consistent failures.

k8s-github-robot · 2018-03-31T04:16:03Z

Automatic merge from submit-queue (batch tested with PRs 61894, 61369). If you want to cherry-pick this change to another branch, please follow the instructions here.

brancz · 2018-12-15T22:41:35Z

Hi, next time when adding metrics please tag sig-instrumentation, as these metrics don't comply with the Kubernetes metrics instrumentation guidelines. Adding these metrics to the metrics overhaul targeted for 1.14 (kubernetes/enhancements#655).

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/M Denotes a PR that changes 30-99 lines, ignoring generated files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Mar 19, 2018

k8s-ci-robot requested review from resouer and sjpotter March 19, 2018 21:49

k8s-ci-robot assigned loburm Mar 19, 2018

loburm reviewed Mar 21, 2018

View reviewed changes

rramkumar1 force-pushed the expose-kubelet-health-checks branch from 365e4a8 to b860cad Compare March 21, 2018 15:35

k8s-ci-robot assigned tallclair Mar 21, 2018

tallclair reviewed Mar 26, 2018

View reviewed changes

rramkumar1 force-pushed the expose-kubelet-health-checks branch 3 times, most recently from a2e4115 to 01d51ba Compare March 30, 2018 17:29

Expose kubelet health checks using new prometheus endpoint

c0c5566

rramkumar1 force-pushed the expose-kubelet-health-checks branch from 01d51ba to c0c5566 Compare March 30, 2018 18:08

tallclair reviewed Mar 30, 2018

View reviewed changes

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label Mar 31, 2018

k8s-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Mar 31, 2018

k8s-github-robot merged commit 20f7f37 into kubernetes:master Mar 31, 2018

brancz mentioned this pull request Aug 7, 2019

Kubernetes Metrics Overhaul kubernetes/enhancements#1206

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Expose kubelet health checks using new prometheus endpoint #61369

Expose kubelet health checks using new prometheus endpoint #61369

rramkumar1 commented Mar 19, 2018 •

edited

Loading

rramkumar1 commented Mar 19, 2018 •

edited

Loading

loburm left a comment

loburm Mar 21, 2018

rramkumar1 Mar 21, 2018

loburm Mar 21, 2018

rramkumar1 Mar 21, 2018

rramkumar1 commented Mar 21, 2018

tallclair Mar 26, 2018

rramkumar1 Mar 26, 2018 •

edited

Loading

loburm Mar 27, 2018

tallclair Mar 28, 2018

rramkumar1 Mar 28, 2018

loburm Mar 29, 2018

rramkumar1 Mar 29, 2018

tallclair left a comment

tallclair Mar 30, 2018

rramkumar1 Mar 30, 2018 •

edited

Loading

tallclair commented Mar 31, 2018

k8s-ci-robot commented Mar 31, 2018

fejta-bot commented Mar 31, 2018

k8s-github-robot commented Mar 31, 2018

brancz commented Dec 15, 2018

Expose kubelet health checks using new prometheus endpoint #61369

Expose kubelet health checks using new prometheus endpoint #61369

Conversation

rramkumar1 commented Mar 19, 2018 • edited Loading

rramkumar1 commented Mar 19, 2018 • edited Loading

loburm left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rramkumar1 commented Mar 21, 2018

Choose a reason for hiding this comment

rramkumar1 Mar 26, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tallclair left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

rramkumar1 Mar 30, 2018 • edited Loading

Choose a reason for hiding this comment

tallclair commented Mar 31, 2018

k8s-ci-robot commented Mar 31, 2018

fejta-bot commented Mar 31, 2018

k8s-github-robot commented Mar 31, 2018

brancz commented Dec 15, 2018

rramkumar1 commented Mar 19, 2018 •

edited

Loading

rramkumar1 commented Mar 19, 2018 •

edited

Loading

rramkumar1 Mar 26, 2018 •

edited

Loading

rramkumar1 Mar 30, 2018 •

edited

Loading