Document steps to expose metrics for EKS-A components

docs/content/en/docs/clustermgmt/observability/expose-metrics.md

@@ -0,0 +1,146 @@
+---
+title: "Expose metrics for EKS-A components"
+linkTitle: "Expose metrics"
+weight: 100
+date: 2024-04-06
+description: >
+  Expose metrics for EKS-A components
+---
+
+Some kubernetes system components like kube-controller-manager, kube-scheduler and kube-proxy expose metrics only on the localhost by default. In order to expose metrics for these components so that other monitoring systems like Prometheus can scrape them, you need to deploy some proxy as a Daemonset on the host network. The proxy pods also need to be configured with control plane tolerations so that they can be scheduled on the control plane nodes.
+
+To configure a proxy for exposing metrics on an EKS-A cluster, one would have to perform the following steps:
+
+1.  Create a new cluster role for the client to access the metrics endpoint of the components.
+    ```bash
+    cat << EOF | kubectl apply -f -
+    apiVersion: rbac.authorization.k8s.io/v1
+    kind: ClusterRole
+    metadata:
+      name: metrics-reader
+    rules:
+      - nonResourceURLs:
+          - "/metrics"
+        verbs:
+          - get
+    EOF
+    ```
+
+2.  Create a new cluster role binding to bind the above cluster role to the client pod's service account.
+    
+    This is an example cluster role binding for the prometheus client.
+    ```bash
+    cat << EOF | kubectl apply -f -
+    apiVersion: rbac.authorization.k8s.io/v1
+    kind: ClusterRoleBinding
+    metadata:
+      name: metrics-reader-binding
+    subjects:
+    - kind: ServiceAccount
+      name: prometheus-server
+      namespace: observability
+    roleRef:
+      kind: ClusterRole
+      name: metrics-reader
+      apiGroup: rbac.authorization.k8s.io
+    EOF
+    ```
+
+3.  Create a config map to store the proxy configuration.
+    
+    This is an example config map for HAProxy.
+    ```bash
+    cat << EOF | kubectl apply -f -
+    apiVersion: v1
+    kind: ConfigMap
+    metadata:
+      name: metrics-proxy
+    data:
+      haproxy.cfg: |
+        defaults
+          mode http
+          timeout connect 5000ms
+          timeout client 5000ms
+          timeout server 5000ms
+          default-server maxconn 10
+
+        frontend kube-proxy
+          bind ${NODE_IP}:10249
+          http-request deny if !{ path /metrics }
+          default_backend kube-proxy
+        backend kube-proxy
+          server kube-proxy 127.0.0.1:10249 check
+
+        frontend kube-controller-manager
+          bind ${NODE_IP}:10257
+          http-request deny if !{ path /metrics }
+          default_backend kube-controller-manager
+        backend kube-controller-manager
+          server kube-controller-manager 127.0.0.1:10257 ssl verify none check
+
+        frontend kube-scheduler
+          bind ${NODE_IP}:10259
+          http-request deny if !{ path /metrics }
+          default_backend kube-scheduler
+        backend kube-scheduler
+          server kube-scheduler 127.0.0.1:10259 ssl verify none check
+    EOF
+    ```
+
+4.  Create a daemonset for the proxy and mount the config map volume onto the proxy pods.
+    
+    This is an example configuration for the HAProxy daemonset.
+    ```bash
+    cat << EOF | kubectl apply -f -
+    apiVersion: apps/v1
+    kind: DaemonSet
+    metadata:
+      name: metrics-proxy
+    spec:
+      selector:
+        matchLabels:
+          app: metrics-proxy
+      template:
+        metadata:
+          labels:
+            app: metrics-proxy
+        spec:
+          tolerations:
+          - key: node-role.kubernetes.io/control-plane
+            operator: Exists
+            effect: NoSchedule
+          hostNetwork: true
+          containers:
+            - name: haproxy
+              image: haproxy:2.9
+              env:
+                - name: NODE_IP
+                  valueFrom:
+                    fieldRef:
+                      apiVersion: v1
+                      fieldPath: status.hostIP
+                - name: TOKEN
+                  value: "$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)"
+              ports:
+                - name: kube-proxy
+                  containerPort: 10249
+                - name: kube-ctrl-mgr
+                  containerPort: 10257
+                - name: kube-scheduler
+                  containerPort: 10259
+              volumeMounts:
+                - mountPath: "/usr/local/etc/haproxy"
+                  name: haproxy-config
+          volumes:
+            - configMap:
+                name: metrics-proxy
+              name: haproxy-config
+    EOF
+    ```
+
+5.  Verify that the metrics are exposed to the client pods by running the following commands:
+    ```bash
+    kubectl exec -it {client-pod-name} -n {client-pod-namespace} -- bash
+    export TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
+    curl -H "Authorization: Bearer ${TOKEN}" http://{node-IP}:{component-port}/metrics
+    ```

docs/content/en/docs/clustermgmt/observability/overview.md

@@ -17,4 +17,5 @@ AWS offers comprehensive monitoring, logging, alarming, and dashboard capabiliti
 1. [Verify EKS Anywhere cluster status]({{< relref "./cluster-verify" >}})
 1. [Use the EKS Connector to view EKS Anywhere clusters and resources in the EKS console]({{< relref "./cluster-connect" >}})
 1. [Use Fluent Bit and Container Insights to send metrics and logs to CloudWatch]({{< relref "./fluentbit-logging" >}})
-1. [Use ADOT to send metrics to AMP and AMG](https://aws.amazon.com/blogs/mt/using-curated-packages-and-aws-managed-open-source-services-to-observe-your-on-premise-kubernetes-environment/)
+1. [Use ADOT to send metrics to AMP and AMG](https://aws.amazon.com/blogs/mt/using-curated-packages-and-aws-managed-open-source-services-to-observe-your-on-premise-kubernetes-environment/)
+1. [Expose metrics for EKS-A components]({{< relref "./expose-metrics" >}})