P7 — Zero-code auto-instrumentation (OTel Operator + Beyla)

[gangster]

Part of the observability epic #102 (Track D). Full design: docs/plans/102-observability-stack.md.

Scope. OpenTelemetry Operator (annotation-driven SDK auto-inject: Java/Python/Node/.NET/Go) + Grafana Beyla (eBPF — RED metrics + traces, no app changes). Dogfood on a hub workload + a canary; becomes the default tenants inherit at P10.

Verify. An un-instrumented sample workload emits RED metrics + traces with zero code change.

Dependency / ordering. Needs P3 (stores). Sequenced before the preprod spoke (P10) so apps inherit instrumentation on onboard.

[gangster]

Approach decided — see ADR-077: Application Instrumentation Strategy. Layered, platform-injection-first (apps don't wire telemetry; the platform injects it, like securityContext / Pod Identity):

• P7a — Beyla (eBPF, zero-code) = the foundation. One DaemonSet → RED metrics + service graph + request-level traces for every workload, any language, existing + new, with no app/template change. Dogfood on the platform cluster first (Backstage/Grafana/Keycloak/ArgoCD serve real HTTP → a non-empty service graph immediately).
• Platform-inject the OTLP endpoint (OTel Operator Instrumentation CR / ns default / Kyverno mutate of OTEL_EXPORTER_OTLP_ENDPOINT) — never app-hardcoded.
• P7b — OTel Operator SDK auto-inject = opt-in enrichment, delivered via the golden-path templates (the inject-<lang> annotation in the scaffolder skeleton) — code-level spans on top of the Beyla baseline. Template instrumentation support lands here, as the richer tier — that's P14 (#591) scope.

Beyla obviates the Tempo metrics-generator for RED/service-graph → folds into P6 (#585) (don't enable the generator).

Sequence: Beyla on platform (dogfood) → endpoint injection → Operator + template opt-in → unblocks P6 APM / P9 SLOs / P13 isolation → P10 spoke traces the first real tenant app (app-alpha-shop).

Cost note: Beyla = ~3 small DaemonSet pods (per-node), and it requires elevated kernel privileges (eBPF) — runs in the already-privileged-PSA observability ns. Roughly net-neutral vs. the metrics-generator it replaces.

Building starts with P7a (the observability-beyla module + platform dogfood) once ADR-077 is agreed.