feat(kuma-cp) golden signals

[lobkovilya]

Summary

Current PR introduces support of Golden Signals: latency, traffic, errors, and saturation.

New dashboard 'Kuma Service'

• Latency

max(histogram_quantile(0.99, rate(envoy_cluster_upstream_rq_time_bucket{kuma_io_services=~".*$service.*",mesh="$mesh",envoy_cluster_name=~"localhost_.*"}[1m])))

• Traffic

  • Incoming

  sum(rate(envoy_cluster_upstream_rq_total{mesh="$mesh",kuma_io_services=~".*$service.*", envoy_cluster_name=~"localhost_.*"}[1m]))
  

  • Outgoing

  sum(rate(envoy_cluster_upstream_rq_total{mesh="$mesh",kuma_io_services=~".*$service.*", envoy_cluster_name!~"localhost_.*", envoy_cluster_name!="kuma_envoy_admin"}[1m]))
  

• Status Codes

  • Incoming

  sum(rate(envoy_cluster_external_upstream_rq_xx{mesh="$mesh",kuma_io_services=~".*$service.*", envoy_cluster_name=~"localhost_.*"}[1m])) by (envoy_response_code_class)
  

  • Outgoing

  sum(rate(envoy_cluster_external_upstream_rq_xx{mesh="$mesh",kuma_io_services=~".*$service.*", envoy_cluster_name!~"localhost_.*", envoy_cluster_name!="kuma_envoy_admin"}[1m])) by (envoy_response_code_class)
  

• CPU

max(sum(rate(container_cpu_usage_seconds_total[1m])) by (namespace, pod) * on (namespace, pod) group_right(kuma_io_service) envoy_server_live{kuma_io_services=~".*$service.*"}) by (dataplane) /
max(sum(kube_pod_container_resource_limits_cpu_cores) by (namespace, pod) * on (namespace, pod) group_right(kuma_io_service) envoy_server_live{kuma_io_services=~".*$service.*"}) by (dataplane)

• Memory Utilization

max(sum(container_memory_working_set_bytes{image!=""}) by (namespace, pod) * on (namespace, pod) group_right(kuma_io_service) envoy_server_live{kuma_io_services=~".*$service.*"}) by (dataplane)

• Memory Saturation

max(sum(container_memory_working_set_bytes) by (namespace, pod) * on (namespace, pod) group_right(kuma_io_service) envoy_server_live{kuma_io_services=~".*$service.*"}) by (dataplane) / max(sum(kube_pod_container_resource_limits_memory_bytes) by (namespace, pod) * on (namespace, pod) group_right(kuma_io_service) envoy_server_live{kuma_io_services=~".*$service.*"}) by (dataplane)

New row on 'Kuma Mesh' dashboard

• Latency

sum(histogram_quantile(0.99, rate(envoy_cluster_upstream_rq_time_bucket{mesh="$mesh",envoy_cluster_name=~"localhost_.*"}[1m]))) by (kuma_io_service)

• Traffic

sum(rate(envoy_cluster_upstream_rq_total{mesh="$mesh",envoy_cluster_name=~"localhost_.*"}[1m])) by (kuma_io_service)

• Status codes

sum(rate(envoy_cluster_external_upstream_rq_xx{mesh="$mesh", envoy_cluster_name=~"localhost_.*"}[1m])) by (kuma_io_service,envoy_response_code_class)

Full changelog

• New charts
• Change the way how we pack the dashboards into ConfigMap because of the annotation size limit

Issues resolved

Fix #XXX

Documentation

• Link to the website documentation PR

[jakubdyszkiewicz]

Thanks for the screenshots, it's helpful to review it.

Kuma Mesh dashboard

• Can we have separate chart for 5xx (and maybe combined with 4xx)? What I want to see as a Mesh operator if I go to mesh dashboard is whether the whole system is OK. If 2xx and 5xx are mixed from all the services it will be hard to distinguish if there is problem or not.
• "Overview" -> "HTTP"?

Kuma Service

• Fix the legend in memory and cpu utilization
• nit: incoming 2xx -> Incoming 2xx
• I'd say - drop the stacks in Kubernetes row. I want to see which instance one is taking the most CPU, same with memory and especially memory saturation going over % is not great
• Second pane on the top, heading of dataplanes. Can it be named Dataplanes not dataplane?
• General -> HTTP?
• What if we look at L4 service? Will we also have L4 metrics here?

[lobkovilya]

@jakubdyszkiewicz thank you for the review.

Kuma Mesh dashboard

• Can we have separate chart for 5xx (and maybe combined with 4xx)? What I want to see as a Mesh operator if I go to mesh dashboard is whether the whole system is OK. If 2xx and 5xx are mixed from all the services it will be hard to distinguish if there is problem or not.

  I agree, but then I'd leave only 5xx (and 4xx) chart because successful status codes are covered by the Traffic chart.

• "Overview" -> "HTTP"?

  Yeah, probably "HTTP" makes more sense because "Latency" and "Status codes" work only for HTTP traffic. But if it's not "Overview" then I don't think it should be the first row of this dashboard. What do you think?

  

Kuma Service

• What if we look at L4 service? Will we also have L4 metrics here?

  We will see everything except "Latency" and "Status codes", is it okay?

  

[jakubdyszkiewicz]

I agree, but then I'd leave only 5xx (and 4xx) chart because successful status codes are covered by the Traffic chart.

ok

Yeah, probably "HTTP" makes more sense because "Latency" and "Status codes" work only for HTTP traffic. But if it's not "Overview" then I don't think it should be the first row of this dashboard. What do you think?

I think it's fine for this to be the first row

We will see everything except "Latency" and "Status codes", is it okay?

What is request/sec in this context? What will this show for redis?

[lobkovilya]

Kuma Mesh dashboard

• Create separate chart Errors Status Codes to aggregate only 5xx and 4xx errors:

  sum(rate(envoy_cluster_external_upstream_rq_xx{mesh="$mesh", envoy_cluster_name=~"localhost_.*", envoy_response_code_class=~"4|5"}[1m])) by (kuma_io_service,envoy_response_code_class)
  

  

• Rename Overview to HTTP

Kuma Service

• Fixed legend:

  

• Rename incoming to Incoming

  

• Get rid of stacks

• Rename dataplane to Dataplanes:

  

• Rename General to HTTP

• if HTTP is treated as TCP then every Request/Response is a new connection, I guess with Redis it works the same way