Observability Learning Cluster

A Kubernetes cluster for learning how observability, identity, and ingress fit together in practice. Started as an OpenTelemetry experiment, grew to cover Flux CD, secret management, network policies, and the Gateway API.

What's in the cluster

                        Browser
                          │
              ┌───────────┴───────────┐
              │  Envoy Gateway        │
              │  (Gateway API)        │
              │  :80 / :443           │
              └───┬──────────────┬────┘
                  │              │
    grafana.127.0.0.1.nip.io   auth.127.0.0.1.nip.io
                  │              │
                  ▼              ▼
              Grafana        Authentik
                  │
                  ▼
              InfluxDB v2 ◄── otel-collector-gateway (Deployment)
                                  │  k8s_cluster receiver
                                  │  OTLP (gRPC :4317 / HTTP :4318)
                                  │
                              otel-collector-agent (DaemonSet)
                                  kubeletstats + filelog per node

Component	What it does	Docs
Envoy Gateway	Routes external traffic into the cluster using the Gateway API	README
OTel Collector	Collects metrics, logs, and traces from the cluster and apps	README
InfluxDB	Time-series database that stores all telemetry	README
Grafana	Dashboards and visualization, with OIDC login via Authentik	README
Identity (Authentik)	SSO provider — handles authentication for Grafana	README
Kyverno	Policy engine — enforces security and best-practice rules at admission time	README
Flux CD	GitOps — the cluster reconciles itself from this repo	README

Quick start

Prerequisites

• kind v0.20+
• kubectl
• helm v3
• Flux CLI
• An age private key for SOPS secret decryption (see Secret management)

Create the cluster

AGE_KEY_FILE=/path/to/keys.txt ./scripts/kind-up.sh

The script creates a 3-node kind cluster (1 control-plane + 2 workers), installs Flux, and applies all manifests. Once reconciliation completes:

• Grafana — http://grafana.127.0.0.1.nip.io
• Authentik — http://auth.127.0.0.1.nip.io

No port-forwarding needed — Envoy Gateway handles ingress via nip.io (a wildcard DNS service where *.127.0.0.1.nip.io resolves to 127.0.0.1).

Tear down

./scripts/kind-down.sh

Force a reconciliation

flux reconcile kustomization scratch-infra --with-source
flux reconcile kustomization scratch --with-source

Secret management

Secrets are encrypted at rest in this repo using SOPS with age encryption. This means the YAML files in git contain encrypted values that Flux decrypts at apply time.

• .sops.yaml controls which files and fields get encrypted
• encrypted_regex: ^(data|stringData)$ ensures only secret values are encrypted — metadata stays readable in diffs
• The age private key lives in a Kubernetes secret (flux-system/sops-age) at runtime, and must be backed up securely

See scripts/bootstrap.sh for how the key is loaded into the cluster.

Repo layout

flux/                          Flux GitRepository + Kustomization
k8s/
  policy/                      Policy engine namespace
    kyverno/                     Kyverno HelmRelease + policies
  ingress/                     Ingress namespace
    envoy-gateway/               Envoy Gateway HelmRelease + Gateway
  observability/               Observability namespace
    otel-collector/              OTel Collector gateway + agent
    influxdb/                    InfluxDB deployment + secret
    grafana/                     Grafana + dashboards + OIDC
  identity/                    Identity namespace
    authentik/                   Authentik server + worker
    postgres/                    PostgreSQL for Authentik
    redis/                       Redis for Authentik
kind/                          kind cluster config
scripts/                       Bootstrap and lifecycle scripts

Component versions

Component	Image / Chart
Kyverno	kyverno/kyverno v3.7.1
Envoy Gateway	envoyproxy/gateway-helm v1.7.1
OTel Collector	otel/opentelemetry-collector-contrib:0.118.0
InfluxDB	influxdb:2.7
Grafana	grafana/grafana:11.4.0
Authentik	ghcr.io/goauthentik/server:2024.10.5
PostgreSQL	postgres:16
Redis	redis:7

Open items

See plan.md for outstanding work and future improvements.