Feature: Add OTEL integration for tracing and metrics

[kusold]

Description

The current observability package only provides request correlation and structured logging. There's no OpenTelemetry integration for distributed tracing or metrics.

Current State

// observability/middleware.go
func CorrelationAndAudit(...) func(http.Handler) http.Handler {
    // ...
    slog.Info("request", fields...)  // Only logging
}

Missing

1. Distributed Tracing

   • HTTP span per request
   • Database span per query
   • OAuth span per external call
   • Trace propagation via headers

2. Metrics

   • Request duration histogram
   • Request count by endpoint/status
   • Active sessions gauge
   • Database connection pool metrics
   • OAuth latency metrics

3. OTEL Configuration

   • Exporter endpoint (OTLP)
   • Sampling configuration
   • Service name/resource attributes

Proposed Implementation

// observability/otel.go
type OTELConfig struct {
    Enabled       bool
    ServiceName   string
    ExporterURL   string // OTLP endpoint
    SampleRate    float64
}

func SetupOTEL(cfg OTELConfig) (func(context.Context) error, error) {
    // Initialize tracer provider
    // Initialize meter provider
    // Return shutdown function
}

// Middleware
func TracingMiddleware(serviceName string) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            ctx, span := otel.Tracer(serviceName).Start(r.Context(), r.URL.Path)
            defer span.End()
            // ...
        })
    }
}

Database Integration

// db/db.go - enhance existing tracing
func setupTracing(cfg *pgxpool.Config) *pgxpool.Config {
    // Add OTEL tracer alongside slog tracer
    cfg.ConnConfig.Tracer = otelpgx.NewTracer()
}

Benefits

• Distributed tracing across services
• Performance insights
• Error tracking with context
• SLO/SLI monitoring

Impact

• Effort: Medium
• Benefit: High (production observability)
• Priority: Medium