prod monitoring has persistent false-positive control-plane alerts and severity noise

[Git-on-my-level]

Summary

Prod monitoring is firing persistent control-plane / scrape / rule alerts while application workloads appear healthy. This reduces trust in alerts during real incidents.

Evidence

Observed via Prometheus/Grafana read-only checks on 2026-07-06 UTC.

Firing alerts included:

• KubeProxyDown critical
• KubeSchedulerDown critical
• KubeControllerManagerDown critical
• PrometheusRuleFailures critical
• TargetDown warnings for kubelet / coredns / gpu-metrics
• Watchdog

At the same time:

• Public API and agent healthchecks returned 200.
• prod-omi-gke pods in prod-omi-backend were 1/1 Running.
• kube_deployment_status_replicas_unavailable was 0 for prod Omi backend services.
• Pending/Failed/Unknown pod counts from Prometheus were 0 for prod backend/monitoring/system namespaces in the sampled query.

Separately, pusher logs include many normal INFO request lines emitted at ERROR severity, which further increases monitoring noise.

Expected behavior

Alerts should represent actionable service or infrastructure degradation. Managed-GKE control-plane scrape limitations and known unavailable targets should be adjusted, suppressed, or relabeled so they do not page as critical app-impacting issues.

Acceptance criteria

• Determine whether the control-plane alerts are invalid for managed GKE scrape topology.
• Fix or suppress invalid KubeProxyDown, KubeSchedulerDown, and KubeControllerManagerDown alerts.
• Fix PrometheusRuleFailures in the affected rule groups.
• Triage TargetDown for kubelet/coredns/gpu-metrics and make the alert actionable or suppress expected gaps.
• Reduce INFO-as-ERROR application log noise, especially pusher request logs.
• Document which prod alerts should page vs remain dashboard-only.

Priority

P2/P3: monitoring reliability issue; not itself a prod outage, but it weakens incident response.