Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system inspired by Prometheus. It is designed to be very cost effective and easy to operate, as it does not index the contents of the logs, but rather a set of labels for each log stream.
Alloy is a flexible, high performance, vendor-neutral distribution of the OpenTelemetry Collector. It’s fully compatible with the most popular open source observability standards such as OpenTelemetry and Prometheus.
Beyla Grafana Beyla uses eBPF to automatically inspect application executables and the OS networking layer, and capture trace spans related to web transactions and Rate Errors Duration (RED) metrics for Linux HTTP/S and gRPC services. All data capture occurs without any modifications to application code or configuration.
Warning
Beyla needs access to various Linux interfaces to instrument applications, load eBPF programs and manage network interface filters. It runs unprivileged here, with all capabilities dropped and eight added back explicitly, plus hostPID, hostNetwork and three host mounts. See docs/beyla.md.
Promtail is an agent which ships the contents of local logs to a private Grafana Loki instance or Grafana Cloud.
Note
Alloy is a replacement for Promtil, it essentially replaces the log collector/scraper that traditionally used Promtail, Grafana Agent or OTel Agent.
Tempo is an open source, easy-to-use, and high-scale distributed tracing backend. Tempo is cost-efficient, requiring only object storage to operate, and is deeply integrated with Grafana, Prometheus, and Loki. Tempo can ingest common open source tracing protocols, including Jaeger, Zipkin, and OpenTelemetry.
Mimir lets you scale metrics to 1 billion active series and beyond, with high availability, multi-tenancy, durable storage, and blazing fast query performance over long periods of time.
Pyroscope is a multi-tenant, continuous profiling aggregation system, aligning its architectural design with Grafana Mimir, Grafana Loki, and Grafana Tempo. This integration enables a cohesive correlation of profiling data with existing metrics, logs, and traces.
I am assuming you are already familiar with Grafana Stack.
- Kubernetes Cluster >= v1.28
- Familiarity with Grafana Stack
- Observability
- ...
- Profit?
┌─────────────────────────────────────────────────────────────────────────────┐
│ GRAFANA DASHBOARD │
│ ┌─────────────────────────────────────────────────────────────────────────┐│
│ │ UNIFIED VISUALIZATION ││
│ │ 📊 Metrics 📝 Logs 🔗 Traces 🔥 Profiles 🚨 Alerts 📈 Dashboards ││
│ └─────────────────────────────────────────────────────────────────────────┘│
└─────────────────────────────────┬───────────────────────────────────────────┘
│ Query APIs
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ DATA SOURCE BACKENDS │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ PROMETHEUS │ │ MIMIR │ │ LOKI │ │ TEMPO │ │
│ │ │ │ │ │ │ │ │ │
│ │ PromQL API │ │ PromQL API │ │ LogQL API │ │ TraceQL API │ │
│ │ /api/v1/ │ │ /api/v1/ │ │ /loki/api/ │ │ /api/v2/ │ │
│ │ │◄─┤ │◄─┤ │◄─┤ │ │
│ │ Short-term │ │ Long-term │ │ Log Aggr. │ │ Distributed │ │
│ │ Metrics │ │ Metrics │ │ & Search │ │ Tracing │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
│ │
│ ┌─────────────┐ │
│ │ PYROSCOPE │ │
│ │ │ │
│ │ Pprof API │◄─────────────────────────────────-┤
│ │ /api/v1/ │ │
│ │ │ │
│ │ Continuous │ │
│ │ Profiling │ │
│ └─────────────┘ │
└─────────────────────────────┬───────────────────────────────────────────────┘
│ Data Collection
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ TELEMETRY AGGREGATION │
│ ┌─────────────────────────────────┐ │
│ │ GRAFANA ALLOY │ │
│ │ │ │
│ │ • OTEL Receiver (4317/4318) │ │
│ │ • Prometheus Scraper (9090) │ │
│ │ • Log Processor & Router │ │
│ │ • Trace Processor & Exporter │ │
│ │ • Profile Collector & Forwarder │ │
│ └─────────────┬───────────────────┘ │
│ │ │
│ ┌──────────────┴───────────────┐ │
│ │ BEYLA (eBPF) │ │
│ │ Auto-instrumentation for │ │
│ │ RED metrics & traces │ │
│ └──────────────────────────────┘ │
└──────────────────────────────────┼──────────────────────────────────────────┘
│ Telemetry Collection
▼
┌─────────────────────────────────────────────────────────────────────────────┐
│ KUBERNETES CLUSTER │
│ │
│ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ ┌─────────────┐ │
│ │ APPLICATION │ │ APPLICATION │ │ APPLICATION │ │OBSERVABILITY│ │
│ │ POD A │ │ POD B │ │ POD C │ │ SERVICES │ │
│ │ │ │ │ │ │ │ │ │
│ │ /metrics │ │ /metrics │ │ /metrics │ │ ConfigMaps │ │
│ │ stdout logs │ │ stdout logs │ │ stdout logs │ │ Services │ │
│ │ OTEL traces │ │ OTEL traces │ │ OTEL traces │ │ Ingress │ │
│ │ pprof/:6060 │ │ pprof/:6060 │ │ pprof/:6060 │ │ RBAC │ │
│ │ │ │ │ │ │ │ Secrets │ │
│ └─────────────┘ └─────────────┘ └─────────────┘ └─────────────┘ │
└─────────────────────────────────────────────────────────────────────────────┘
VISUALIZATION QUERIES:
======================
📊 Metrics Query: Grafana → PromQL → Prometheus/Mimir
📝 Logs Query: Grafana → LogQL → Loki
🔗 Traces Query: Grafana → TraceQL→ Tempo
🔥 Profiles Query: Grafana → Pprof → Pyroscope
DATA COLLECTION FLOW:
=====================
Applications → Beyla/Alloy → Storage Backends → Grafana Dashboards
GRAFANA DATASOURCES:
====================
┌─────────────────┬─────────────────┬──────────────────────────────────────┐
│ DATASOURCE │ QUERY LANGUAGE │ ENDPOINT │
├─────────────────┼─────────────────┼──────────────────────────────────────┤
│ Prometheus │ PromQL │ http://prometheus-server │
│ Mimir │ PromQL │ http://mimir/prometheus │
│ Loki │ LogQL │ http://loki:3100 │
│ Tempo │ TraceQL │ http://tempo:3100 │
│ Pyroscope │ Pprof │ http://pyroscope:4040 │
└─────────────────┴─────────────────┴──────────────────────────────────────┘
kubectl apply -k ./deploymentThis will deploy the following services in monitoring namespace:
| Service | Description |
|---|---|
| Grafana | Preconfigured with Prometheus, Mimir, Loki, Tempo, Pyroscope |
| Prometheus | Metrics collection and short-term storage |
| Mimir | Long-term metrics storage |
| Loki | Log aggregation and storage |
| Alloy | Telemetry collector, pushes to Prometheus, Mimir, Loki, Tempo, Pyroscope |
| Beyla | eBPF-based auto-instrumentation, pushes to Alloy |
| Agent which ships the contents of local logs to Grafana Loki | |
| Pyroscope | Continuous profiling backend |
| Tempo | Distributed tracing backend |
Once deployed, access Grafana UI via:
kubectl port-forward -n monitoring svc/grafana 3000:3000Note
If you deploy to different namespace, update ClusterRoleBinding subject namespace to match
Grafana open and composable observability stack
Beyla gives you spans with no code changes, which is the fastest way to get traces flowing. It cannot propagate trace context across a process boundary, though, so traces stop at each service and Tempo's service graph can only draw virtual nodes for the peers it infers.
For NodeJS services, @saidsef/tracing-node covers that gap. It wraps the OpenTelemetry SDK with HTTP, fetch/undici, Express, Elasticsearch, IORedis, AWS SDK and Pino instrumentation, and registers the W3C Trace Context propagator, so a call from one service to another arrives as a child span in the same trace. That is what turns Tempo's service graph from a list of virtual nodes into real service-to-service edges.
npm install @saidsef/tracing-node --saveimport { setupTracing } from '@saidsef/tracing-node';
setupTracing({serviceName: 'my-service', url: 'http://alloy:4317'});Point url at the Alloy OTLP gRPC receiver deployed here, which forwards to Tempo. The two approaches compose - Beyla for services you cannot change, the SDK for the ones you can.
This is an attempt to demystify the different components of LGTM+ Stack, deploying the full stack can seem overwhelming, breaking it down to smaller composable pieces will hopefully help you better understand each service and its configuration.
Our latest and greatest source of *grafana-loki-on-k8s can be found on GitHub, Fork us!
We would ❤️ you to contribute by making a pull request.
Please read the official Contribution Guide for more information on how you can contribute.