Skip to content

fix(tracing): wire OTLP pipeline to Jaeger, remove dead ObservabilityConfig#2890

Merged
bug-ops merged 2 commits intomainfrom
2881-otlp-tracing-not-wired
Apr 11, 2026
Merged

fix(tracing): wire OTLP pipeline to Jaeger, remove dead ObservabilityConfig#2890
bug-ops merged 2 commits intomainfrom
2881-otlp-tracing-not-wired

Conversation

@bug-ops
Copy link
Copy Markdown
Owner

@bug-ops bug-ops commented Apr 11, 2026

Summary

  • Wire TelemetryBackend::Otlpbuild_otlp_layer creates a gRPC OTLP exporter (tonic), a SdkTracerProvider with BatchSpanProcessor + ratio sampler, and registers it globally via tracing-opentelemetry
  • Add OtlpGuard to TracingGuards that calls provider.shutdown() on drop for graceful span flush
  • Fix Cargo.toml: opentelemetry-otlp + opentelemetry_sdk had default-features = false with no features — added grpc-tonic,trace and trace respectively
  • Remove dead ObservabilityConfig struct and all 9 stale [observability] references across config templates and doc comments
  • Update testing.toml: replace [observability] with [telemetry] enabled=true backend="otlp" pointing at localhost:4317

Root causes fixed (closes #2881)

Three root causes confirmed by investigation:

  1. init_tracing reads TelemetryConfig ([telemetry]), not ObservabilityConfig ([observability]) — existing config was targeting the wrong key
  2. build_chrome_layer returned None for any non-Local backend — Otlp was never handled
  3. opentelemetry-otlp had default-features = false with no explicit features, making the gRPC transport unavailable at compile time even if code had existed

Validation

  • 8151 tests pass (up from 8149; 2 new tests added)
  • cargo clippy --lib --bins --features full --workspace -- -D warnings — clean
  • cargo +nightly fmt --check — clean
  • New tests: build_otlp_layer_sample_rate_out_of_range_is_clamped (CI, no network), tracing_guards_drop_with_otel_provider_does_not_panic (CI, no-op provider)
  • #[ignore] live pipeline test available for manual verification with Jaeger running

Follow-up issues

  • P3: OTel internal logger for dropped-span warnings is not bridged to tracing — filed separately
  • M1 security: attribute filter for PII in OTLP spans — future iteration
  • otlp_headers_vault_key vault wiring — out of scope for this PR, warns when set but unused

@bug-ops bug-ops added bug Something isn't working enhancement New feature or request P2 High value, medium complexity labels Apr 11, 2026
@github-actions github-actions Bot added documentation Improvements or additions to documentation rust Rust code changes core zeph-core crate dependencies Dependency updates config Configuration file changes size/L Large PR (201-500 lines) labels Apr 11, 2026
@bug-ops bug-ops enabled auto-merge (squash) April 11, 2026 13:13
bug-ops added 2 commits April 11, 2026 15:14
…Config

`TelemetryBackend::Otlp` was a stub — `build_chrome_layer` returned `None`
for any non-Local backend, and opentelemetry-otlp/opentelemetry_sdk had no
features enabled so the gRPC transport could not compile anyway.

Changes:
- add `build_otlp_layer` behind `#[cfg(feature = "otel")]`: SpanExporter
  (tonic gRPC) → SdkTracerProvider (BatchSpanProcessor + ratio sampler +
  service_name resource) → global::set_tracer_provider; returns provider
  handle for graceful flush on drop
- add `otel_provider: Option<SdkTracerProvider>` to `TracingGuards` with
  explicit `Drop` that calls `provider.shutdown()` to flush pending spans
- clamp `sample_rate` to `[0.0, 1.0]` with `tracing::warn!` on out-of-range
- warn when `otlp_headers_vault_key` is set but not yet wired
- fix Cargo.toml: opentelemetry-otlp gets `["grpc-tonic","trace"]`,
  opentelemetry_sdk gets `["trace"]`; tonic 0.12/0.14 coexistence documented
- remove dead `ObservabilityConfig` struct and all re-exports; purge all
  stale `[observability]` sections from config templates and doc comments
- update testing.toml: replace `[observability]` with correct `[telemetry]`
  block pointing at localhost:4317
- fix docker-compose.tracing.yml comments to reference `[telemetry]`

Closes #2881
@bug-ops bug-ops force-pushed the 2881-otlp-tracing-not-wired branch from c5ca06c to 7763684 Compare April 11, 2026 13:14
@bug-ops bug-ops merged commit becd50b into main Apr 11, 2026
30 checks passed
@bug-ops bug-ops deleted the 2881-otlp-tracing-not-wired branch April 11, 2026 13:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

bug Something isn't working config Configuration file changes core zeph-core crate dependencies Dependency updates documentation Improvements or additions to documentation enhancement New feature or request P2 High value, medium complexity rust Rust code changes size/L Large PR (201-500 lines)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

OTLP tracing backend not wired — Jaeger receives zero spans

1 participant