Phase 3.5: Observability (Prometheus, structured logging, health checks)

[renecannao]

Phase 3.5 — Observability

Goal: Modern observability stack — Prometheus metrics, structured JSON logging, and Kubernetes-ready health check endpoints.

Sub-tasks

Prometheus Metrics

• 3.5.1 Add Prometheus client library (prometheus/client_golang)
• 3.5.2 Expose /metrics endpoint with standard Go runtime metrics
• 3.5.3 Add discovery metrics: orchestrator_instances_discovered, orchestrator_discovery_latency_seconds, orchestrator_discovery_errors_total
• 3.5.4 Add replication metrics: orchestrator_replication_lag_seconds, orchestrator_replication_broken_total
• 3.5.5 Add recovery metrics: orchestrator_recoveries_total, orchestrator_recovery_duration_seconds, orchestrator_recovery_errors_total
• 3.5.6 Add cluster metrics: orchestrator_clusters_total, orchestrator_cluster_instances
• 3.5.7 Add raft metrics (if raft mode): orchestrator_raft_leader, orchestrator_raft_peers, orchestrator_raft_applied_index
• 3.5.8 Maintain backward compatibility with existing Graphite metrics (go/metrics/)
• 3.5.9 Configuration: enable/disable Prometheus, custom endpoint path

Structured Logging

• 3.5.10 Evaluate logging libraries (slog from stdlib, zerolog, or zap)
• 3.5.11 Implement structured logger wrapper supporting both text and JSON output
• 3.5.12 Add context fields to log entries (instance key, cluster name, recovery ID)
• 3.5.13 Replace golib/log calls with structured logger (incremental migration)
• 3.5.14 Configuration: log format (text/json), log level, output destination

Health Check Endpoints

• 3.5.15 /health/live — liveness probe (process is running)
• 3.5.16 /health/ready — readiness probe (backend DB connected, discovery running)
• 3.5.17 /health/leader — raft leadership status (for raft deployments)
• 3.5.18 Kubernetes deployment example with probe configuration
• 3.5.19 Documentation: Grafana dashboard JSON, alerting rules examples

Design considerations

• Prometheus metrics must have low cardinality — avoid per-instance labels where possible, use summary/histogram types
• Structured logging should be opt-in (default to existing text format for backward compatibility)
• Health endpoints should be lightweight (no DB queries for liveness)
• Consider existing /api/health and /api/leader-check endpoints — extend, don't duplicate

Dependencies

None — can start independently.

Part of Phase 3 — Feature Direction.