Observability Enterprise (Kubernetes + OpenTelemetry + Grafana)

Production-inspired Kubernetes observability stack for an ASP.NET Core Web API with OpenTelemetry and the Grafana ecosystem.

Architecture

• Traces: ASP.NET Core OTLP -> Grafana Alloy (gateway) -> Tempo -> Grafana
• Logs:
  • App OTLP logs -> Alloy -> Loki -> Grafana
  • Pod logs -> Alloy (daemonset, Kubernetes log tailing) -> Loki -> Grafana
• Metrics: ASP.NET Core OTLP metrics -> Alloy -> Prometheus (remote_write receiver enabled) -> Grafana

Repo Layout

• src/Observability.Enterprise.Api: ASP.NET Core Web API (.NET 8)
• deploy/environments/local-kind: demo-friendly values + Kustomize overlay
• deploy/environments/cluster: stricter defaults for a generic cluster
• deploy/scripts: kind + deploy helpers
• dashboards/grafana: Grafana dashboards

Prerequisites

• Docker
• kubectl
• Helm
• kind
• kustomize (kubectl ships it)

Local kind setup

make kind-up
make deploy
make port-forward

Grafana (local-kind): http://localhost:3000 (admin/admin)

Generate traffic

curl -s http://localhost:8080/healthz
curl -s http://localhost:8080/api/hello
curl -s "http://localhost:8080/api/work?ms=200"
curl -s http://localhost:8080/api/error

# Load test
for i in $(seq 1 50); do
  curl -s "http://localhost:8080/api/work?ms=$((RANDOM % 400))" >/dev/null
  curl -s http://localhost:8080/api/error >/dev/null || true
done

Explore data in Grafana

• Metrics: Dashboards -> "API RED"
• Logs: Explore -> Loki -> {namespace="observability-enterprise"}
• Traces: Explore -> Tempo -> search by service observability-enterprise-api

Port-forward (manual)

kubectl -n observability-enterprise port-forward svc/kube-prometheus-stack-grafana 3000:80
kubectl -n observability-enterprise port-forward svc/kube-prometheus-stack-prometheus 9090:9090
kubectl -n observability-enterprise port-forward svc/tempo 3200:3200
kubectl -n observability-enterprise port-forward svc/observability-enterprise-api 8080:8080

Cluster environment

kubectl apply -k deploy/environments/cluster --enable-helm

• Update deploy/environments/cluster/api-deployment.yaml with your registry image.
• Update deploy/environments/cluster/values/kube-prometheus-stack.values.yaml with a secure Grafana password.

Troubleshooting

• Alloy gateway logs:

  kubectl -n observability-enterprise logs deploy/alloy-gateway

• Alloy agent logs:

  kubectl -n observability-enterprise logs daemonset/alloy-agent

• ServiceMonitor targets:

  kubectl -n observability-enterprise get servicemonitor

• OTLP endpoint connectivity from the API pod:

  kubectl -n observability-enterprise exec deploy/observability-enterprise-api -- curl -s http://alloy-gateway:4318

Notes

• Prometheus remote_write receiver is enabled so Alloy can push OTLP-converted metrics directly into Prometheus.
• App metrics, logs, and traces are all sent to Alloy using OTLP over HTTP (http://alloy-gateway:4318).