Export readiness and liveness metrics

[okke-formsma]

/kind feature
Currently there is no way to monitor liveness and readiness probe failures using kube-state-metrics. I would very much like to be able to see these for every pod.

[brancz]

The kubelet has the /metrics/probes path, which exposes the results of probes. Given that's where the metrics originate that's also a better place that kube-state-metrics 🙂 .

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[fejta-bot]

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten

[fejta-bot]

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

[k8s-ci-robot]

@fejta-bot: Closing this issue.

In response to this:

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[krish7919]

The kubelet has the /metrics/probes path, which exposes the results of probes. Given that's where the metrics originate that's also a better place that kube-state-metrics 🙂 .

@brancz Can you please elaborate on this? We are currently using Datadog with our k8s cluster and the dd-agent does not seem to get this metric.

[brancz]

I know nothing about the datadog agent so I can’t say why. I see them in Prometheus :)

[sravanthi1993]

I know nothing about the datadog agent so I can’t say why. I see them in Prometheus :)

@brancz: Can you please share how to configure prometheus.yml to get these probe metrics ?

[adamjk-dev]

+1 @brancz Can you elaborate on how you are finding the liveness/readiness probes via Promtheus? I have the same interest and am not finding the metrics as is. Maybe this depends on the version of k8s/kube-state-metrics/prometheus or the configuration. Thanks!

[sb1975]

/remove-lifecycle stale
/reopen

This is a good requirement which is needed to be available to show on Prometheus Dashboard using Grafana. Couldnt find any Grafana dashboard or Prometheus command to show this.

/help-needed

[k8s-ci-robot]

@sb1975: You can't reopen an issue/PR unless you authored it or you are a collaborator.

In response to this:

/remove-lifecycle stale
/reopen

This is a good requirement which is needed to be available to show on Prometheus Dashboard using Grafana. Couldnt find any Grafana dashboard or Prometheus command to show this.

/help-needed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[brancz]

These metrics already exist on the kubelet endpoint /metrics/probes, which I think is where it belongs.

[purplehat7]

@brancz What are the metric names in Prometheus that you're seeing?

[janotav]

@acannon828 and others who might stumble upon this thread searching for the same:

prober_probe_total

[ghost]

this is a probe sample in prometheus-configmap.yaml:

    - job_name: 'kubelet-probes'
      kubernetes_sd_configs:
        - role: node
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      relabel_configs:
        - action: labelmap
          regex: __meta_kubernetes_node_label_(.+)
        - target_label: __address__
          replacement: kubernetes.default.svc:443
        - source_labels: [__meta_kubernetes_node_name]
          regex: (.+)
          target_label: __metrics_path__
          replacement: /api/v1/nodes/${1}/proxy/metrics/probes

then youcan find prober_probe_total in Prometheus