Request metrics and traffic

Request metrics and traffic

CPU and memory say how busy a workload is; they do not say whether it answers well. Sextant shows request metrics — how many requests a workload gets, how many fail and how slow they are — and a traffic graph of who calls whom, for clusters that already collect such metrics in Prometheus: an Istio or Linkerd mesh, ingress-nginx, Grafana Beyla, Cilium Hubble or Caretta.

Kubernetes itself has no request metrics, and Sextant installs nothing to get them: it reads what your mesh, ingress controller or eBPF agent already reports to Prometheus (or VictoriaMetrics), with fixed queries of its own. Nothing in the cluster is changed.

Contents: The Requests section · CPU throttling · The traffic graph · Sources and what they need · When there is nothing to show · Limits · Privacy and safety

The Requests section

The details of a Deployment, StatefulSet, DaemonSet or Pod have a Requests section, right after the CPU and memory charts:

• Request rate (requests per second over the last 5 minutes), Error rate (the share of failed requests) and Latency as p50, p95 and p99;
• two charts of the last hour: requests per second and failed requests per second;
• the source the numbers come from ("From Istio"), with a line saying what exactly it counts — for example "Only requests that came through ingress-nginx to the Services that select these pods."

When several sources report the workload (Istio and Hubble on a Cilium cluster, say), each gets its own block; Sextant never adds them up, since they usually see the same requests.

What counts as an error depends on the source: an HTTP 5xx answer, or a gRPC status other than OK; for Linkerd, Linkerd's own verdict (classification="failure", which also covers requests with no answer). An error rate is given only when there were requests ("No requests to count errors of."). A latency that cannot be computed is unknown, never 0; a source without a latency histogram says so.

The numbers of a workload are the pods its controller made: Sextant recognises them by their names (web-7d4b9c6f5-x2k8p for a Deployment web, db-0 for a StatefulSet db), as Right-sizing does, so a pod of web-api is never counted for web. Where the source itself names the workload (Istio, Hubble), its own label is used.

The section is read when it is shown and refreshed once a minute while it stays open. For a Pod or workload in a Linkerd mesh with linkerd-viz installed, the Service mesh section points here with Show request metrics.

CPU throttling

Below the request metrics, CPU throttling shows, per container, the share of CPU periods in which it was throttled over the last hour (from the cAdvisor counters container_cpu_cfs_throttled_periods_total and container_cpu_cfs_periods_total, which most Prometheus installations scrape). This part needs Prometheus but no request source.

When a container with a CPU limit was throttled in 25 % of its periods or more, Sextant says the limit is the likely cause of slow responses, and names the CPU limit Right-sizing suggests from its usage history — or, until right-sizing has enough history, points to its section.

The traffic graph

Cluster → Map has two modes: Objects (the Resource map) and Traffic. The traffic mode draws the calls between workloads with an end in the namespaces of the namespace filter: callers on the left, the services they call to the right, one column per hop.

• Namespaces: at most the first 5 of the filter (the object map keeps 3); with All namespaces the page asks you to pick.
• Window: Last 5 minutes (refreshed every 30 seconds) or Last hour (every minute).
• Lines: a line is as thick as its request rate and coloured by its error rate — under 1 %, 1–5 %, 5 % or more, or dashed when the error rate is not known. Two ends seen by two sources are one line, as thick as the busier source (Istio and Hubble see the same calls; they are not added up). TCP connections per second are a different unit: they are shown beside the rate, and a pair seen only as TCP connections is drawn thinnest. TCP bytes per second (from Caretta) are a third unit: they are shown beside the rate as bytes per second, filed under the TCP filter, and never change a line's thickness.
• Filters: namespace, protocol (HTTP, gRPC, TCP) and status (with failed requests, with 5xx, 4xx or 2xx answers). They only change the drawing.
• A box (a workload, a Service, or an unknown end) is a button that lists its callers and callees. A line opens the call: rate, error rate, latency p50 and p95, answers by status class, protocol and source; Open opens the end's details (Sextant looks the name up as a Deployment, StatefulSet, then DaemonSet, or a Service), and Requests of opens its Requests section.
• List shows the same calls as a table, busiest first — the way to read the graph with the keyboard or a screen reader.

An end the source cannot name — a caller outside the mesh, a callee outside the cluster, a pod without the labels the source needs — is drawn as Outside the mesh or unlabelled (one per namespace when the source tells the namespace), never dropped. Its panel says why there is nothing to open.

The graph shows the 200 busiest calls; when there are more the page says "Showing the 200 busiest of N". If your Prometheus does not answer some of the detail queries (error rates, answers by status, latency), the calls are still drawn and the page says which details could not be read.

The graph is read only while the Traffic mode is shown and the window is visible; a refresh is skipped while the previous one is still running.

Sources and what they need

Sextant uses the Prometheus or VictoriaMetrics of the cluster's settings (Find automatically, a manual address, or Off; see Settings and its VictoriaMetrics section). Once per connection (kept for 5 minutes) it asks that Prometheus which of a short, fixed list of metric names it has, and decides the sources from them. When that Prometheus has no Linkerd metrics, the Prometheus that linkerd-viz installs (linkerd-viz/prometheus:9090) is asked too.

Source	Metrics Sextant looks for	Requests section	Traffic graph	What to make sure of
Istio	istio_requests_total, istio_request_duration_milliseconds_bucket; istio_tcp_connections_opened_total for TCP	Yes, from the receiving proxy	Yes: HTTP, gRPC and TCP	The proxies' metrics are scraped (Istio's Prometheus add-on or your own scrape config).
Linkerd	response_total together with response_latency_ms_bucket; tcp_open_total for TCP	Yes, from the receiving proxy	Yes: HTTP, gRPC and TCP	linkerd-viz is installed, or your Prometheus scrapes the Linkerd proxies.
ingress-nginx	nginx_ingress_controller_requests, nginx_ingress_controller_request_duration_seconds_bucket	Yes, only the requests that came through the ingress	No: it does not say who sent a request	The controller's metrics are scraped (a ServiceMonitor or PodMonitor works).
Grafana Beyla	http_server_request_duration_seconds_*, rpc_server_duration_seconds_*; for the graph traces_service_graph_request_server_seconds_*, traces_service_graph_request_client_seconds_*, traces_service_graph_request_failed_total	Yes	Only with its service graph metrics	Kubernetes metadata is on (labels k8s_namespace_name, k8s_pod_name); for the graph, the application_service_graph metrics feature is on.
Cilium Hubble	hubble_http_requests_total, hubble_http_request_duration_seconds_bucket	Yes	Yes: HTTP
Caretta	caretta_links_observed	No	Yes: TCP bytes only	Caretta (groundcover's eBPF agent) runs on the nodes and its metrics are scraped.

Notes:

• Caretta sees TCP connections, not requests: there are no errors, status classes or latency, and it is never a source of the Requests section. It gives the bytes each connection sent, so the graph shows bytes per second, both directions summed where both ends' nodes run Caretta (one direction only where just one end's node does). The page says once that Caretta sees connections only. Bytes are ranked apart from requests and connections, so they never crowd request edges out of the 200 busiest calls.
• A connection through a ClusterIP shows as two lines. With kube-proxy the client's socket is connected to the Service's address while the server's socket sees the backing pod, so Caretta reports client → Service (only what the client sent) and client → workload (only what the workload answered). Sextant keeps them apart and labels the first TCP to a Service (ClusterIP); it does not fold the Service onto its pods, since a Service can select pods of several workloads or none. Ends Caretta cannot name (node, external) are drawn as the unknown end.
• Anything that exports the OpenTelemetry HTTP and RPC server metrics with Beyla's Kubernetes labels counts as Beyla.
• Hubble reports a call at the callee only with ingress L7 visibility and at the caller only with egress visibility. Sextant counts the callee's report, and the caller's only for the pairs the callee does not report, so a call is never counted twice. Without labelsContext naming the workloads Hubble has no series of a workload: "no data", not an error. Hubble's L4 flow metrics count a flow at both ends, so Hubble gives HTTP calls only.
• Beyla's service graph is read from the server-side and client-side histograms, not from traces_service_graph_request_total, which Beyla 2.0–2.2 count twice for a call between two instrumented services.
• ingress-nginx is matched through the Services that select the workload's pods, which Sextant reads itself (at most 20). When those cannot be read, ingress-nginx is reported as unreadable, not as absent.
• Label names are fixed. A scrape configuration that renames pod or namespace (for example to kubernetes_pod_name) gives "no data for these pods" for the sources matched by pod; older Beyla releases that label http_status_code instead of http_response_status_code show no errors.
• The rights needed are the same as for the other Prometheus views: get on services/proxy of the Prometheus Service; for ingress-nginx also reading the workload and the namespace's Services.

When there is nothing to show

Message	Meaning
"…none is set up or found for this cluster."	No Prometheus. Open Cluster settings → Metrics and set its address (see Settings).
"Prometheus has no request metrics…" (Requests) or "Prometheus has no metrics that name both ends of a call…" (Traffic)	Prometheus is there, but none of the sources reports to it. The page lists what would provide the data. On the Traffic page, sources that give requests without callers (ingress-nginx, Beyla without its service graph) are named, with a pointer to the Requests section.
"Prometheus has metrics of …, but none for these pods."	A source exists, but not for this workload: it is not in the mesh, its traffic does not pass the ingress, or the labels the source needs are missing.
"… saw no calls with an end in these namespaces in the last 5 minutes."	The sources work, but nothing in these namespaces called or was called in the window. Try Last hour.
"Could not read the metrics of …"	Prometheus did not answer for that source (a timeout, a limit). Its numbers are unknown, not zero.
"The query for … was too long for a proxy in front of Prometheus."	See Troubleshooting.

Limits

• The numbers come from Prometheus (or VictoriaMetrics), so they are as fresh and as complete as its scrapes (VictoriaMetrics answers about 30 seconds behind by default): rates are over 5 minutes (or the hour on the graph), latencies are histogram estimates.
• Requests section: one request per source for the current values and one for the last hour, plus one for CPU throttling, once a minute; at most 50 containers in the throttling table.
• Traffic graph: at most 5 namespaces, 200 calls (the busiest), 4000 series and 8 MiB per Prometheus answer (a bigger answer is an error, never cut short), 18 seconds per query.
• Every query is kept short enough for a proxy in front of Prometheus (under 6 KiB of URL with five namespaces of the longest names). A Prometheus set by its URL gets a query again as a POST if it answers 414 or 431.
• What the source does not see is not on the page: traffic that bypasses the mesh or the ingress, or pods Beyla or Hubble do not observe.

Privacy and safety

• Nothing is installed or enabled. Sextant never installs a mesh, Beyla, Hubble, linkerd-viz or Prometheus, and never changes their settings; without a source the page only says what would provide one.
• Fixed queries. Every PromQL query is built in Sextant's backend from a fixed template per source. What the window sends is only a cluster, a namespace, a kind and a name (or up to 5 namespaces and a window), each checked to be a valid Kubernetes name and escaped again in the query; nothing you type becomes part of a query.
• Read-only. Only your Prometheus or VictoriaMetrics is asked, through the API server's service proxy (or the address you set), plus a look-up by name when you click Open on the graph. Nothing is written to the cluster.
• Window only. Request metrics and the traffic graph are not available to AI agents or extensions: there is no MCP tool and no extension method for them.

Next: Resource map, Right-sizing, Service mesh, Traffic diagnosis.