Monitoring

[BehRoooz]

Summary

This PR introduces and refines the monitoring stack from monitoring into main to improve observability, alerting, and operational debugging across CAFA-5 services and pipelines.

Changes

• Updated Prometheus scrape configuration in monitoring/prometheus.yml
• Added/refined alert rules in monitoring/alerts.yml
• Added/updated Grafana datasource provisioning in monitoring/grafana/provisioning/datasources/prometheus.yml
• Added/updated Grafana dashboard provider config in monitoring/grafana/provisioning/dashboards/providers.yml
• Added/updated dashboards:
  • monitoring/grafana/dashboards/cafa5-domain-pipelines.json
  • monitoring/grafana/dashboards/cafa5-service-health.json
• Expanded documentation in monitoring/README.md (setup, usage, troubleshooting)

Why

• Improve real-time visibility into pipeline and service behavior
• Enable faster detection of regressions and runtime failures
• Standardize local/dev monitoring setup for contributors
• Reduce mean time to resolution through actionable dashboards and alerts

Validation

• Monitoring stack starts and services are discoverable
• Prometheus loads scrape and alerting configs successfully
• Grafana provisions datasource and dashboards on startup
• Key panels and alert rules show expected metric wiring

Reviewer Notes

• Dashboard JSON is large/generated; review panel intent and metric queries
• Alert thresholds may require environment-specific tuning after merge

Risk

Low to moderate. Changes are isolated to monitoring configuration, dashboards, and docs, but misconfigured thresholds or queries could produce noisy alerts or missing panels.

Rollback

Revert this PR to restore previous monitoring configuration and dashboards.