helm/kagent: declarative OTLP headers + service name + resource attributes (unblocks Langfuse Cloud / Honeycomb / Datadog / ...)

[nirzOps]

Related issue: #2126
Related draft PR: #2128

The itch

otel.tracing in helm/kagent/values.yaml lets you enable OTLP tracing and point at any endpoint, but the moment that endpoint is authenticated (Langfuse Cloud, Honeycomb, Datadog, Grafana Cloud, New Relic, Chronosphere, ...), the chart has no way to plumb the auth header. It also can't set OTEL_SERVICE_NAME or OTEL_RESOURCE_ATTRIBUTES, which is how the OTEL spec expects service.name, deployment.environment.name, tenant/team/env tags to flow into resource attributes.

The Go controller in go/core/internal/telemetry/tracing.go already understands all of these env vars via autoexport.NewSpanExporter(ctx) and resource.WithFromEnv(). This is a pure chart-surface gap.

Why post this as a Discussion

CONTRIBUTING.md asks for large-ish changes to go through issue + Slack + agreed plan before a draft PR. The issue is the "what and why"; this discussion is the "shape of the API" — the exact values.yaml keys are the design surface I'd like feedback on before anything gets merged.

Concrete Langfuse example (the thing I hit in the wild)

Sending kagent controller traces to Langfuse Cloud requires:

• Endpoint: https://cloud.langfuse.com/api/public/otel/v1/traces — supported today.
• Protocol: http/protobuf — supported today via otel.tracing.exporter.otlp.protocol.
• Auth header: Authorization: Basic <base64(publicKey:secretKey)> — not supported.
• Resource attribute: langfuse.environment=<env> so traces are partitioned per env — not supported.

Today the only way to get all four is to bypass the chart's otel.* config entirely and wire env vars into the controller Pod via controller.envFrom from a hand-written ConfigMap + Secret. Two things get worse with that approach:

1. otel.tracing.enabled in values.yaml is effectively a lie — the truthful value lives in a sidecar ConfigMap.
2. Anyone writing wrapper charts for kagent has to duplicate the same pattern (which is what happened at Elementor: elementor/elementor-cloud-external-charts@1e8b9436 + elementor/elementor-agents@13545eb).

Proposed shape

Four new declarative knobs, all optional, all backward compatible:

otel:
  serviceName: ""             # -> OTEL_SERVICE_NAME
  resourceAttributes: {}      # map -> OTEL_RESOURCE_ATTRIBUTES
  tracing:
    exporter:
      otlp:
        headers: {}           # map -> OTEL_EXPORTER_OTLP_TRACES_HEADERS
  logging:
    exporter:
      otlp:
        headers: {}           # map -> OTEL_EXPORTER_OTLP_LOGS_HEADERS

• Maps serialize to key1=value1,key2=value2 with sorted keys (same pattern already used for DEFAULT_AGENT_POD_LABELS).
• Nothing is rendered when a field is unset → helm template output is byte-identical for existing installs.
• Secret headers stay outside the schema. The existing controller.envFrom hook is exactly the right extension point for a Basic-auth header stored in a Secret; the chart's data.OTEL_EXPORTER_OTLP_TRACES_HEADERS is transparently overridden by the Pod-level env var. The values.yaml comments document this.

End-to-end Langfuse install after the change

# values.yaml
otel:
  serviceName: kagent
  resourceAttributes:
    deployment.environment.name: production
    langfuse.environment: production
    team: platform
  tracing:
    enabled: true
    exporter:
      otlp:
        endpoint: https://cloud.langfuse.com/api/public/otel/v1/traces
        protocol: http/protobuf
        insecure: false

controller:
  envFrom:
    - secretRef:
        name: langfuse-auth   # Secret with OTEL_EXPORTER_OTLP_HEADERS=Authorization=Basic <...>

That's the entire configuration — no sidecar ConfigMap/Secret, no wrapper chart required, and the same shape works for Honeycomb / Datadog / Grafana Cloud / New Relic just by swapping the header Secret and endpoint.

Open design questions

1. Native headersSecretRef? Should we also add otel.tracing.exporter.otlp.headersSecretRef: {name, key} that renders as an env valueFrom.secretKeyRef on the controller Deployment? Pro: single-schema declarative install. Con: duplicates controller.envFrom, adds a template branch. I default to no for the first cut — happy to add if you want.
2. Symmetric logging.exporter.otlp.headers? Included in my draft. Cheap and consistent with the existing schema (both signals expose insecure/timeout). Speak up if you'd rather keep logs out of scope for round one.
3. Location of serviceName / resourceAttributes. I put them at otel.* root because per OTEL spec they are global resource attributes shared across signals. Alternative would be otel.tracing.* only, but that ignores logs. Preferences?
4. Any appetite to also patch helm/tools/querydoc/templates/configmap.yaml in the same PR? It has an identical gap. My instinct is no, separate PR, but I'll defer.

What's ready

• Chart change on nirzOps/kagent:feat/otel-headers-service-name-resource-attrs (~60 line diff).
• helm/kagent/tests/controller-configmap_test.yaml covering backward compat + each new field + full Langfuse render (11 tests, all pass).
• helm lint clean, all existing 225 unit tests still pass.
• DCO-signed commit ready.

Draft PR link will be added here as soon as it's opened. Grateful for any pushback on the API shape before I un-draft it.

[nirzOps]

Draft PR is up: #2128. Only chart-surface change, ~60 line diff, backward compatible, 11 new unit tests. Grateful for feedback on the four values.yaml keys before I un-draft.