Page about monitoring Kubernetes with Sensu

[hillaryfraley]

Description

Adds a resource page for monitoring Kubernetes with Sensu, which covers Kubernetes data sources, Sensu + Prometheus for monitoring Kubernetes, the recommended approach (sidecar), and existing Kubernetes integrations (from Bonsai).

Links to Sensu resources with more information, like blog posts and whitepapers.

Motivation and Context

Closes #3256

[calebhailey]

Capturing some notes from a discussion w/ @jspaleta here... It sounds like we need to organize this content to address one or more of the following three topics (in priority order):

1. What is supported? (sensu-agent daemonset, sensu-agent sidecar)

   • Elevated privileges: why & how? (example)
   • Suggested environment variables (e.g. KUBE_HOST, or syncing K8s namespaces w/ Sensu namespace via Downward API)
   • What isn't supported: running sensu-backend in Kubernetes (tl;dr: it works, but it's not a supported platform from a commercial support SLA perspective)

2. How to monitor Kubernetes w/ Sensu?

   • TODO: add a template for https://github.com/sensu/sensu-kubernetes-events (see webinar for more context/content here)
   • Collecting metrics w/ Sensu (see also: output_metric_tags, which add value here)
     • Kubelet metrics (multiple)
     • Cadvisor metrics
     • Kube-state-metrics (cluster-level metrics)
   • TBD how many of these metrics are gauge style metrics that we could use w/ metric thresholds; i.e. which of these are collection only vs monitoring + alerting???

3. How to monitor Kubernetes workloads?

   • Example agent sidecar healthcheck (e.g. HTTP)
   • Example app container emitting events to agent sidecar events API

If we cover topic number 1 above, I think that would make a sufficient MVP here. The second topic may require additional effort.

[hillaryfraley]

Content removed from the draft (adding here so I don't have to dig back through commits later):

Use the Sensu catalog of Kubernetes health and metrics checks

Kubelet host metrics collection

Prometheus can collect and analyze data on your Kubernetes deployment, but the data model is constrained: data must be represented as a measurement and can lack context as a result, and exporters provide only summarized data and scrape only periodically.
Use Sensu with Prometheus to get a complete picture of your Kubernetes deployment.

Sensu's [kubelet host metrics collection][10] monitor allows you to collect host metrics, including for [kubelet][20] hosts, with the [Prometheus Node Exporter][11].
The Sensu kubelet host metrics collection collects metrics but does not provide alerts and requires Prometheus Node Exporter.

Learn more about using Sensu with Prometheus to monitor Kubernetes in [Monitoring Kubernetes + Docker, part 3: Sensu + Prometheus][5].

Kubelet monitor

The Sensu [Kubelet monitor][12] includes both health and metrics checks:

• [Kubelet health check][13]
• [Kubelet etcd health check][14]
• [Kubelet metrics][15]
• [Kubelet probe metrics][16]
• [Kubelet cAdvisor metrics][17]

The metrics checks collect metrics but do not provide alerts.

The Sensu kubelet monitor health checks and metrics collectors rely on [kubelet][20] metrics and cAdvisor as data sources.

Kubelet metrics provide data for the Kubernetes nodes and the jobs they run.
Metrics components include [kube-apiserver][19], [kube-controller-manager][22], and [kube-scheduler][21].

The built-in kubelet cAdvisor collects, aggregates, processes, and exports metrics for each running container.
cAdvisor can track resource isolation parameters and historical resource usage so Kubernetes can designate how much memory is being used.

Kubernetes cluster metrics

The [Kubernetes cluster metrics][18] monitor collects kubelet metrics from the [metrics-server][23] API.

Kubernetes cluster metrics requires [kube-state-metrics][24], which listens to the Kubernetes API server and provides high-level information about a Kubernetes cluster.
kube-state-metrics data include which containers are running, their current state, the number of contains in a particular state, and whether any are unhealthy or at capacity.

Kubernetes cluster metrics collects metrics but does not provide alerts.

Sensu Kubernetes Events Check

{{% notice note %}}
NOTE: The Sensu Kubernetes Events Check is an alpha release of a Sensu-native integration and may include breaking changes.
{{% /notice %}}

The [Sensu Kubernetes Events Check][25] is an Sensu Check that uses the [Kubernetes Event API][26] to identify events that should generate corresponding Sensu events.
The check itself returns an OK status (exit code 0) unless it encounters a problem like failing to authenticate with Kubernetes for API access.
For each matching event type the check finds, it creates separate events with the Sensu [agent API][7].

Use our curated, configurable [quick-start template][27] to get started with the Sensu Kubernetes Events Check plugin and integrate Sensu with your existing workflows.

Read [Filling gaps in Kubernetes observability with the Sensu Kubernetes Events integration][8] for more information about the Kubernetes Events API and the Sensu Kubernetes Events Check integration.

Links:

https://github.com/prometheus/node_exporter
https://kubernetes.io/docs/concepts/workloads/pods/
https://sensu.io/blog/monitoring-kubernetes-part-4-the-sensu-native-approach
https://sensu.io/resources/whitepaper/whitepaper-monitoring-kubernetes-the-sidecar-pattern
https://sensu.io/blog/monitoring-kubernetes-docker-part-3-sensu-prometheus
https://bonsai.sensu.io/assets/sensu/sensu-kubernetes-events
../agent/#create-observability-events-using-the-agent-api
https://sensu.io/blog/filling-gaps-in-kubernetes-observability-with-the-sensu-kubernetes-events-integration
https://bonsai.sensu.io/assets/sensu-plugins/sensu-plugins-kubernetes
https://github.com/sensu/catalog/blob/main/monitors/node_exporter/node_exporter.yaml
https://prometheus.io/docs/guides/node-exporter/
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml#L37-L53
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml#L55-L72
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml#L74-L97
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml#L99-L122
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kubelet.yaml#L124-L147
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/kube-state-metrics.yaml
hhttps://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/
https://kubernetes.io/docs/reference/command-line-tools-reference/kubelet/
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/
https://kubernetes.io/docs/reference/command-line-tools-reference/kube-controller-manager/
https://github.com/kubernetes-sigs/metrics-server
https://github.com/kubernetes/kube-state-metrics
https://bonsai.sensu.io/assets/sensu/sensu-kubernetes-events
https://v1-18.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.18/#event-v1-core
https://github.com/sensu/catalog/blob/main/monitors/kubernetes/events/sensu-kubernetes-events.yaml

[hillaryfraley]

Also removed:

From sidecar section
To monitor Kubernetes itself, add a Sensu agent on all Kubernetes hosts.

[hillaryfraley]

@jspaleta I updated the draft to fit the MVP description in item 1. in Caleb's comment.

[hillaryfraley]

On hold pending Sensu Catalogs work