Feature Request: Move from Monitoring Platform to Observability Platform

[ricsanfre]

Feature Request

Improve current Monitoring platform, based on Prometheus (metrics) and EFK (logs) to a Observability Platform adding traces monitoring and a single plane of glass.

Explore solution based on Loki (logs), Tempo (traces), Prometheus (metrics) and Grafana as single plane of glass for monitoring.

Main requirements

• Loki as complement of the current EFK platform, not a replacement. ES is used mainly for Log Analytics (log content is completely indexed) while Loki can be used for Observability (only log labels are indexed) having together logs, metrics and traces in the same Grafana Dashboards.

• Current logs collection/distrution layer based on fluentbit/fluentd should be used to feed logs to ES and to Loki, instead of deploying a separate collector (Loki promtail)

  Fluentbit/Fluentd Loki plugins are available, so current forwarder/aggregator architecture could be maintained just adding Loki as backend for logs.

  • Fluentbit Loki output plugins
  • Fluentd Loki output plugin plugin maintained by Grafana community

• Linkerd distributed tracing mechanism should be enabled. See this Grafana's blog entry where the process is explained.

Issue Progress

• Add Loki
• Reconfigure Fluentbit/Fluentd to send logs to ES and Loki
• Add Grafana Tempo
• Configure Linkerd distributed tracing
• Configure, Ingress Controller, Traefik traces

[ricsanfre]

References:

• Setting up Open Source Observability Stack on Kubernetes from scratch: https://k8staz.com/setting-up-open-source-observability-stack-on-kubernetes-from-scratch
• Kubernetes observability with Loki, Cortex and Tempo
• To test distributed tracing solution (Tempo), hotrod application can be used.
• Traefik and open tracing (https://wshihadeh.medium.com/traefik-2-and-open-tracing-da5c483486e3)

[ricsanfre]

About Traefik tracing

• Activate tracing using B3 propagation protocol (only one supported by Linkerd).

  --tracing.zipkin=true
  --tracing.zipkin.httpEndpoint=http://zipkin:9411/api/v2/spans
  --tracing.zipkin.sameSpan=true
  --tracing.zipkin.id128Bit=true
  --tracing.zipkin.sampleRate=1
  

• In order to correlate logs with traces in Grafana, Traefik access log should be configured so, trace ID is also present as a field in the logs. By default no header is included in Traefik's access log. See https://doc.traefik.io/traefik/observability/access-logs/#limiting-the-fieldsincluding-headers