feat(observability): direct-to-cloud OTLP — TLS + auth headers (no sidecar)

[jfwoods]

Status

Docs portion shipped in PR #116 — docs/configuration.md now has a full OTel section covering WH_OTEL_ADDR and every other env var, plus an explicit note on the WithInsecure() constraint and the sidecar workaround. docs/deployment.md covers the SigNoz/Honeycomb (OTLP-native) and Grafana Cloud / Alloy (Prometheus scrape + OTLP traces via Alloy) wiring patterns.

This issue is staying open and re-scoped to the actual feature work that the original docs note pointed at: making WaveHouse able to ship OTLP directly to TLS-protected cloud endpoints without an intermediate sidecar.

Problem (re-scoped)

internal/observability/provider.go calls otlptracegrpc.WithInsecure() (and the equivalents for metrics + logs). That means WaveHouse can ship OTLP only to plaintext receivers — typically a local OTel collector or Grafana Alloy on 127.0.0.1:4317. Operators who want to ship directly to:

• Grafana Cloud's OTLP gateway (https://otlp-gateway-prod-us-east-0.grafana.net/otlp — TLS + Basic auth)
• Honeycomb (https://api.honeycomb.io:443 — TLS + x-honeycomb-team header)
• Datadog OTLP endpoint (TLS + API key)
• Any cloud OTLP endpoint that requires TLS or auth headers

…can't. The standard workaround is a sidecar collector that receives plaintext on localhost and re-exports with TLS + auth configured locally. Annoying for operators running on Fly/Railway/Cloud Run / etc. where adding a sidecar is non-trivial.

Proposed Solution

Two related additions:

1. Endpoint scheme sniffing — if otel.addr starts with https://, configure the gRPC exporters with TLS credentials (grpc.WithTransportCredentials(credentials.NewTLS(...))) instead of WithInsecure(). If it starts with http://, behave as today. If no scheme, keep current behavior (plaintext) for backward compat.
2. Headers env var — WH_OTEL_HEADERS (yaml otel.headers) accepting comma-separated key=value pairs (or repeated env var pattern), wired to otlptracegrpc.WithHeaders(map[string]string) etc. on all three exporters.

Together these unlock WH_OTEL_ADDR=https://otlp-gateway-prod-us-east-0.grafana.net/otlp WH_OTEL_HEADERS=Authorization=Basic\ <base64> working end-to-end — no sidecar.

Alternatives Considered

• Separate otel.tls.enabled boolean — explicit but more knobs than necessary. Scheme sniffing is the standard OTel-SDK pattern (some language SDKs do this natively).
• A per-signal endpoint override (different addr for traces vs metrics vs logs) — Grafana Cloud actually does this (different gateway URLs per signal). Worth scoping in: maybe otel.{traces,metrics,logs}.addr overrides the top-level otel.addr when set. Lower priority; can be follow-up.
• mTLS (client cert auth) — niche; defer until requested.

Additional Context

• Docs from this PR (feat(observability): add otel pipeline and system tracing #116) explicitly call out the limitation in docs/configuration.md so operators aren't surprised, and link to this issue.
• Original issue (pre-rescoping) asked for just the docs change — that part is done. Re-scoping rather than closing+filing-new because the conversation is here.