-
Notifications
You must be signed in to change notification settings - Fork 14k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
update resource metrics pipeline section
- Loading branch information
1 parent
45061ef
commit 10e7039
Showing
2 changed files
with
171 additions
and
47 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
216 changes: 170 additions & 46 deletions
216
content/en/docs/tasks/debug-application-cluster/resource-metrics-pipeline.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,76 +1,200 @@ | ||
| --- | ||
| reviewers: | ||
| - fgrzadkowski | ||
| - piosz | ||
| title: Resource metrics pipeline | ||
| - piosz | ||
| title: Resource metrics pipeline | ||
| content_type: concept | ||
| --- | ||
|
|
||
| <!-- overview --> | ||
|
|
||
| Resource usage metrics, such as container CPU and memory usage, | ||
| are available in Kubernetes through the Metrics API. These metrics can be accessed either directly | ||
| by the user with the `kubectl top` command, or by a controller in the cluster, for example | ||
| Horizontal Pod Autoscaler, to make decisions. | ||
| For Kubernetes, the _Metrics API_ offers a basic set of metrics to support automatic scaling and similar use cases. | ||
| This API makes information available about resource usage for node and pod, including metrics for CPU and memory. | ||
| If you deploy the Metrics API into your cluster, clients of the Kubernetes API can then query for this information, and | ||
| you can use Kubernetes' access control mechanisms to manage permissions to do so. | ||
|
|
||
| <!-- body --> | ||
| The [HorizontalPodAutoscaler](/docs/tasks/run-application/horizontal-pod-autoscale/) (HPA) and [VerticalPodAutoscaler](https://github.com/kubernetes/autoscaler/tree/master/vertical-pod-autoscaler#readme) (VPA) use data from the metrics API to adjust workload replicas and resources to meet customer demand. | ||
|
|
||
| ## The Metrics API | ||
| You can also view the resource metrics using the [`kubectl top`](https://kubernetes.io/docs/reference/generated/kubectl/kubectl-commands#top) command. | ||
|
|
||
| Through the Metrics API, you can get the amount of resource currently used | ||
| by a given node or a given pod. This API doesn't store the metric values, | ||
| so it's not possible, for example, to get the amount of resources used by a | ||
| given node 10 minutes ago. | ||
| {{< note >}} | ||
| The Metrics API, and the metrics pipeline that it enables, only offers the minimum | ||
| CPU and memory metrics to enable automatic scaling using HPA and / or VPA. | ||
| If you would like to provide a more complete set of metrics, you can complement | ||
| the simpler Metrics API by deploying a second | ||
| [metrics pipeline](/docs/tasks/debug-application-cluster/resource-usage-monitoring/#full-metrics-pipeline) | ||
| that uses the _Custom Metrics API_. | ||
| {{< /note >}} | ||
|
|
||
| The API is no different from any other API: | ||
|
|
||
| - it is discoverable through the same endpoint as the other Kubernetes APIs under the path: `/apis/metrics.k8s.io/` | ||
| - it offers the same security, scalability, and reliability guarantees | ||
| Figure 1 illustrates the architecture of the resource metrics pipeline. | ||
|
|
||
| {{< mermaid >}} | ||
| flowchart RL | ||
| subgraph cluster[Cluster] | ||
| direction RL | ||
| S[ <br><br> ] | ||
| A[Metrics-<br>Server] | ||
| subgraph B[Nodes] | ||
| direction TB | ||
| D[cAdvisor] --> C[kubelet] | ||
| E[Container<br>runtime] --> D | ||
| E1[Container<br>runtime] --> D | ||
| P[pod data] -.- C | ||
| end | ||
| L[API<br>server] | ||
| W[HPA] | ||
| C ---->|Summary<br>API| A -->|metrics<br>API| L --> W | ||
| end | ||
| L ---> K[kubectl<br>top] | ||
| classDef box fill:#fff,stroke:#000,stroke-width:1px,color:#000; | ||
| class W,B,P,K,cluster,D,E,E1 box | ||
| classDef spacewhite fill:#ffffff,stroke:#fff,stroke-width:0px,color:#000 | ||
| class S spacewhite | ||
| classDef k8s fill:#326ce5,stroke:#fff,stroke-width:1px,color:#fff; | ||
| class A,L,C k8s | ||
| {{< /mermaid >}} | ||
|
|
||
| Figure 1. Resource Metrics Pipeline | ||
|
|
||
| The architecture components, from right to left in the figure, consist of the following: | ||
|
|
||
| * [cAdvisor](https://github.com/google/cadvisor) - Daemon for collecting, aggregating and exposing container metrics included in Kubelet. cAdvisor reads metrics from cgroups allowing out of the box support for Docker. Note that non-container runtimes need to support [Container Metrics RPS](https://github.com/kubernetes/community/blob/master/contributors/devel/sig-node/cri-container-stats.md) for metrics to be available. | ||
| * [kubelet](https://kubernetes.io/docs/concepts/overview/components/#kubelet) - Node agent for managing container resources. Resource metrics are accessible using the `/metrics/resource` and `/stats` kubelet API endpoints. | ||
| * [Summary API](#summary-api-source) - Kubelet API for discovering and retrieving per-node summarized stats available through the `/stats` endpoint. | ||
| * [metrics-server](#metrics-server) - Cluster addon component that collects and aggregates resource metrics pulled from each kubelet. The K8s API server serves up the Metric API endpoints for use by HPA, VPA and the `kubectl top` command. | ||
| * [Metrics API](#metrics-api) - Kubernetes API supporting access to CPU and memory used for workload autoscaling. Metrics Server is the default implementation provided with many popular K8s distributions, however it can be replaced by alternative adapters based on your preferred monitoring solution. | ||
|
|
||
| <!-- body --> | ||
|
|
||
| The API is defined in [k8s.io/metrics](https://github.com/kubernetes/metrics/blob/master/pkg/apis/metrics/v1beta1/types.go) | ||
| repository. You can find more information about the API there. | ||
| ## Metrics API | ||
|
|
||
| The metrics-server implements the Metrics API. This API allows you to access CPU and memory usage for the nodes and pods in your cluster. Its primary role is to feed resource usage metrics to K8s autoscaler components. | ||
|
|
||
| Here is an example of the Metrics API request for a `minikube` node piped through `jq` for easier reading: | ||
| ```shell | ||
| kubectl get --raw "/apis/metrics.k8s.io/v1beta1/nodes/minikube" | jq '.' | ||
| ``` | ||
|
|
||
| Here is the same API call using `curl`: | ||
| ```shell | ||
| curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/nodes/minikube | ||
| ``` | ||
| Sample reply: | ||
| ```json | ||
| { | ||
| "kind": "NodeMetrics", | ||
| "apiVersion": "metrics.k8s.io/v1beta1", | ||
| "metadata": { | ||
| "name": "minikube", | ||
| "selfLink": "/apis/metrics.k8s.io/v1beta1/nodes/minikube", | ||
| "creationTimestamp": "2022-01-27T18:48:43Z" | ||
| }, | ||
| "timestamp": "2022-01-27T18:48:33Z", | ||
| "window": "30s", | ||
| "usage": { | ||
| "cpu": "487558164n", | ||
| "memory": "732212Ki" | ||
| } | ||
| } | ||
| ``` | ||
| Here is an example of the Metrics API request for a `kube-scheduler-minikube` pod contained in the `kube-system` namespace and piped through `jq` for easier reading: | ||
|
|
||
| ```shell | ||
| kubectl get --raw "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube" | jq '.' | ||
| ``` | ||
| Here is the same API call using `curl`: | ||
| ```shell | ||
| curl http://localhost:8080/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube | ||
| ``` | ||
| Sample reply: | ||
| ```json | ||
| { | ||
| "kind": "PodMetrics", | ||
| "apiVersion": "metrics.k8s.io/v1beta1", | ||
| "metadata": { | ||
| "name": "kube-scheduler-minikube", | ||
| "namespace": "kube-system", | ||
| "selfLink": "/apis/metrics.k8s.io/v1beta1/namespaces/kube-system/pods/kube-scheduler-minikube", | ||
| "creationTimestamp": "2022-01-27T19:25:00Z" | ||
| }, | ||
| "timestamp": "2022-01-27T19:24:31Z", | ||
| "window": "30s", | ||
| "containers": [ | ||
| { | ||
| "name": "kube-scheduler", | ||
| "usage": { | ||
| "cpu": "9559630n", | ||
| "memory": "22244Ki" | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| The Metrics API is defined in the [k8s.io/metrics](https://github.com/kubernetes/metrics) repository. You must enable the [API aggregation layer](/docs/tasks/extend-kubernetes/configure-aggregation-layer/) and register an [APIService](/docs/reference/kubernetes-api/cluster-resources/api-service-v1/) for the `metrics.k8s.io` API. | ||
|
|
||
| To learn more about the Metrics API, see [resource metrics API design](https://github.com/kubernetes/design-proposals-archive/blob/main/instrumentation/resource-metrics-api.md), the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server) and the [resource metrics API](https://github.com/kubernetes/metrics#resource-metrics-api). | ||
|
|
||
|
|
||
| {{< note >}} You must deploy the metrics-server or alternative adapter that serves the Metrics API to be able to access it. {{< /note >}} | ||
|
|
||
| ## Measuring resource usage | ||
|
|
||
| {{< note >}} | ||
| The API requires the metrics server to be deployed in the cluster. Otherwise it will be not available. | ||
| {{< /note >}} | ||
| ### CPU | ||
|
|
||
| ## Measuring Resource Usage | ||
| CPU is reported as the average core usage measured in cpu units. One cpu, in Kubernetes, is equivalent to 1 vCPU/Core for cloud providers, and 1 hyper-thread on bare-metal Intel processors. | ||
|
|
||
| ### CPU | ||
| This value is derived by taking a rate over a cumulative CPU counter provided by the kernel (in both Linux and Windows kernels). The time window used to calculate CPU is shown under window field in Metrics API. | ||
|
|
||
| CPU is reported as the average usage, in | ||
| [CPU cores](/docs/concepts/configuration/manage-resources-containers/#meaning-of-cpu), | ||
| over a period of time. This value is derived by taking a rate over a cumulative CPU counter | ||
| provided by the kernel (in both Linux and Windows kernels). | ||
| The kubelet chooses the window for the rate calculation. | ||
| To learn more about how Kubernetes allocates and measures CPU resources, see [meaning of CPU](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-cpu). | ||
|
|
||
| ### Memory | ||
|
|
||
| Memory is reported as the working set, in bytes, at the instant the metric was collected. | ||
| In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. | ||
| However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. | ||
| It includes all anonymous (non-file-backed) memory since Kubernetes does not support swap. | ||
| The metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim such pages. | ||
| Memory is reported as the working set, measured in bytes, at the instant the metric was collected. | ||
|
|
||
| In an ideal world, the "working set" is the amount of memory in-use that cannot be freed under memory pressure. However, calculation of the working set varies by host OS, and generally makes heavy use of heuristics to produce an estimate. | ||
|
|
||
| The Kubernetes model for a container's working set expects that the container runtime counts anonymous memory associated with the container in question. The working set metric typically also includes some cached (file-backed) memory, because the host OS cannot always reclaim pages. | ||
|
|
||
| To learn more about how Kubernetes allocates and measures memory resources, see [meaning of memory](/docs/concepts/configuration/manage-compute-resources-container/#meaning-of-memory). | ||
|
|
||
| ## Metrics Server | ||
|
|
||
| [Metrics Server](https://github.com/kubernetes-sigs/metrics-server) is a cluster-wide aggregator of resource usage data. | ||
| By default, it is deployed in clusters created by `kube-up.sh` script | ||
| as a Deployment object. If you use a different Kubernetes setup mechanism, you can deploy it using the provided | ||
| [deployment components.yaml](https://github.com/kubernetes-sigs/metrics-server/releases) file. | ||
| The metrics-server fetches resource metrics from the kubelets and exposes them in the Kubernetes API server through the Metrics API for use by the HPA and VPA. You can also view these metrics using the `kubectl top` command. | ||
|
|
||
| The metrics-server uses the Kubernetes API to track nodes and pods in your cluster. The metrics-server queries each node over HTTP to fetch metrics. The metrics-server also builds an internal view of pod metadata, and keeps a cache of pod health. That cached pod health information is available via the extension API that the metrics-server makes available. | ||
|
|
||
| For example with an HPA query, the metrics-server needs to identify which pods fulfill the label selectors in the deployment. | ||
|
|
||
| The metrics-server calls the [kubelet](/docs/reference/command-line-tools-reference/kubelet/) API to collect metrics from each node. Depending on the metrics-server version it uses: | ||
| * Metrics resource endpoint `/metrics/resource` in version v0.6.0+ or | ||
| * Summary API endpoint `/stats/summary` in older versions | ||
|
|
||
|
|
||
| Metrics Server collects metrics from the Summary API, exposed by | ||
| [Kubelet](/docs/reference/command-line-tools-reference/kubelet/) on each node, and is registered with the main API server via | ||
| [Kubernetes aggregator](/docs/concepts/extend-kubernetes/api-extension/apiserver-aggregation/). | ||
| To learn more about the metrics-server, see the [metrics-server repository](https://github.com/kubernetes-sigs/metrics-server). | ||
|
|
||
| Learn more about the metrics server in | ||
| [the design doc](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md). | ||
| You can also check out the following: | ||
|
|
||
| ### Summary API Source | ||
| The [Kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at node, volume, pod and container level, and emits their statistics in | ||
| * [metrics-server design](https://github.com/kubernetes/community/blob/master/contributors/design-proposals/instrumentation/metrics-server.md) | ||
| * [metrics-server FAQ](https://github.com/kubernetes-sigs/metrics-server/blob/master/FAQ.md) | ||
| * [metrics-server known issues](https://github.com/kubernetes-sigs/metrics-server/blob/master/KNOWN_ISSUES.md) | ||
| * [metrics-server releases](https://github.com/kubernetes-sigs/metrics-server/releases) | ||
| * [Horizontal Pod Autoscaling](/docs/tasks/run-application/horizontal-pod-autoscale/) | ||
|
|
||
| ### Summary API source | ||
|
|
||
| The [kubelet](/docs/reference/command-line-tools-reference/kubelet/) gathers stats at the node, volume, pod and container level, and emits this information in | ||
| the [Summary API](https://github.com/kubernetes/kubernetes/blob/7d309e0104fedb57280b261e5677d919cb2a0e2d/staging/src/k8s.io/kubelet/pkg/apis/stats/v1alpha1/types.go) | ||
| for consumers to read. | ||
|
|
||
| Pre-1.23, these resources have been primarily gathered from [cAdvisor](https://github.com/google/cadvisor). However, in 1.23 with the | ||
| introduction of the `PodAndContainerStatsFromCRI` FeatureGate, container and pod level stats can be gathered by the CRI implementation. | ||
| Note: this also requires support from the CRI implementations (containerd >= 1.6.0, CRI-O >= 1.23.0). | ||
| Here is an example of a Summary API request for a `minikube` node: | ||
|
|
||
|
|
||
| ```shell | ||
| kubectl get --raw "/api/v1/nodes/minikube/proxy/stats/summary" | ||
| ``` | ||
| Here is the same API call using `curl`: | ||
| ```shell | ||
| curl http://localhost:8080/api/v1/nodes/minikube/proxy/stats/summary | ||
| ``` |