feat: basic request tracing with jaeger ui for dev

[illegalprime]

basic visualization with jaeger for request timings, nothing deeper at the moment:

[github-actions]

🔐 Codex Security Review

Note: This is an automated security-focused code review generated by Codex.
It should be used as a supplementary check alongside human review.
False positives are possible - use your judgment.

Scope summary

• Reviewed pull request diff only (a2f3e8ee1a521b70e65906d7ed9308096a4a1d1e...cc95c7855f97c9145d38866a1b316d7e25f65a20, exact PR three-dot diff)
• Model: gpt-5.4

💡 Click "edited" above to see previous reviews for this PR.

---

Review Summary

Overall Risk: MEDIUM

Findings

[MEDIUM] HTTP-level tracing will misclassify failed Connect/gRPC streams as successful

• Category: gRPC
• Location: server/internal/handlers/middleware/telemetry.go:15
• Description: The new middleware wraps every RPC with otelhttp and only sees the outer HTTP response code. In this server, that wrapper sits in front of Connect handlers, and Connect servers also speak gRPC, including streaming. For gRPC, the RPC outcome is communicated in trailers when the call closes, not in the outer HTTP status. That means once a stream has started, later connect.CodeInternal, cancellation, or send failures can still be exported as an apparently successful 200 span. (connectrpc.com)
• Impact: Trace-based debugging and alerting will under-report failures on the platform’s core streaming flows, which makes auth failures, cancellations, and backend send errors materially harder to detect during incident response.
• Recommendation: Instrument at the Connect/gRPC layer instead of only at the HTTP layer, so span status comes from connect.Error/RPC status. If you keep the HTTP middleware, exclude or separately sample long-lived streaming RPCs.

[LOW] The new telemetry containers are not fully hardened

• Category: Infrastructure
• Location: server/docker-compose.yaml:94
• Description: The new otel-collector and jaeger services are referenced by tag only, and the collector bind-mounts ./otel-collector-config.yaml without :ro. Docker documents that digests are immutable while tags are not, and that bind mounts are read-write by default unless explicitly marked read-only. In practice, that gives any compromised or retagged collector image a host write path back into the checked-out config file. (docs.docker.com)
• Impact: This is limited to environments that run the compose stack, but on a developer workstation or CI runner it creates a supply-chain/persistence path that can tamper with local telemetry configuration.
• Recommendation: Pin both images by digest and mount ./otel-collector-config.yaml:/etc/otelcol-contrib/config.yaml:ro.

Notes

• The behavior-changing portion of this diff is almost entirely the new telemetry wiring; I did not find new auth, SQL injection, or command-execution issues in the changed hunks.
• No tests were added for TelemetryMiddleware or fleet_telemetry.Setup.
• I could not run Go tests in this environment because the filesystem is read-only.

---

_{Generated by Codex Security Review |
Triggered by: @illegalprime |
Review workflow run}

[ankitgoswami]

maybe rename tracing to observability since we'd also want to use some of these same configs for metrics.