A complete observability stack for development purposes using OpenTelemetry, Grafana, Prometheus, Loki, and Tempo. This stack provides comprehensive monitoring, logging, and tracing capabilities for your applications.
This stack consists of the following components:
- OpenTelemetry Collector - Receives telemetry data from applications and routes it to appropriate backends
- Prometheus - Metrics collection and storage
- Loki - Log aggregation and storage
- Tempo - Distributed tracing backend
- Grafana - Visualization and dashboards for all telemetry data
-
Start the stack:
docker-compose up -d
-
Access Grafana:
- URL: http://localhost:8030
- Username:
admin - Password:
admin
-
Configure your application to send telemetry data to the OpenTelemetry Collector:
- OTLP gRPC:
localhost:4317 - OTLP HTTP:
localhost:4318
- OTLP gRPC:
| Service | Port | Description |
|---|---|---|
| Grafana | 8030 | Web UI for dashboards and visualization |
| OpenTelemetry Collector | 4317, 4318 | OTLP gRPC and HTTP receivers |
| Prometheus | 9090 | Metrics storage (internal) |
| Loki | 3100 | Log storage (internal) |
| Tempo | 3200, 4317, 4318 | Trace storage and OTLP receivers (internal) |
The collector is configured to:
- Receive telemetry data via OTLP (gRPC and HTTP)
- Route traces to Tempo
- Route logs to Loki
- Route metrics to Prometheus
- Export metrics for Prometheus scraping
Pre-configured data sources:
- Prometheus (default) - For metrics visualization
- Loki - For log exploration
- Tempo - For trace analysis
- Cross-correlation - Traces can be linked to logs and metrics
- Local storage backend for development
- 1-hour block retention
- Metrics generation enabled
- Remote write to Prometheus for trace metrics
- File system storage
- TSDB index for better performance
- 24-hour index period
// Example Go application
import (
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
)
// Configure OTLP exporter
exporter, err := otlptracegrpc.New(ctx,
otlptracegrpc.WithEndpoint("localhost:4317"),
otlptracegrpc.WithInsecure(),
)// Example Go application
import (
"go.opentelemetry.io/otel/exporters/prometheus"
)
// Configure Prometheus exporter
exporter, err := prometheus.New()// Example Go application
import (
"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc"
)
// Configure OTLP log exporter
exporter, err := otlploggrpc.New(ctx,
otlploggrpc.WithEndpoint("localhost:4317"),
otlploggrpc.WithInsecure(),
)- Metrics: Create dashboards using Prometheus queries
- Logs: Use Loki data source for log exploration
- Traces: Use Tempo data source for distributed tracing
- Correlation: Click on trace spans to view related logs and metrics
Access Prometheus directly at http://localhost:9090 for advanced metric queries.
- Access Grafana at http://localhost:8030
- Create new dashboards or import existing ones
- Use the pre-configured data sources
Modify otel/otel-collector-config.yaml to:
- Add new receivers (e.g., Jaeger, Zipkin)
- Configure additional processors
- Add new exporters
This configuration is optimized for development. For production:
- Use persistent storage volumes
- Configure proper retention policies
- Set up authentication and TLS
- Use external storage backends (S3, GCS, etc.)
Stop and remove all containers:
docker-compose down -v- OpenTelemetry Documentation
- Grafana Documentation
- Prometheus Documentation
- Loki Documentation
- Tempo Documentation
- Port conflicts: Ensure ports 8030, 4317, 4318 are available
- Data not appearing: Check that your application is sending data to the correct endpoints
- Grafana login issues: Default credentials are admin/admin
View container logs:
docker-compose logs -f [service-name]docker-compose down -v
docker system prune -f
docker-compose up -d