Skip to content

feat: add Prometheus, Grafana, Uptime Kuma, and management proxy#8

Merged
mcansem merged 1 commit into
mainfrom
phase/v0.7.0-observability-stack
Jul 21, 2026
Merged

feat: add Prometheus, Grafana, Uptime Kuma, and management proxy#8
mcansem merged 1 commit into
mainfrom
phase/v0.7.0-observability-stack

Conversation

@mcansem

@mcansem mcansem commented Jul 21, 2026

Copy link
Copy Markdown
Owner

Summary

  • Add docker/observability/: Prometheus, Alertmanager, Grafana, Uptime Kuma. Prometheus/Alertmanager stay internal-only (neither has meaningful auth); Grafana/Uptime Kuma get a public URL through the new management proxy
  • Prometheus joins monitoring_agent_net to scrape node-exporter/cadvisor by container name - their 127.0.0.1-published ports aren't reachable from inside another container, so same-host scraping goes over a shared Docker network instead
  • Three default alert rules (host down, disk >85%, container missing) with a no-op Alertmanager receiver - a real notification channel needs deployment-specific credentials that don't belong in this repo
  • Grafana's Prometheus datasource is provisioned as code; dashboards are a documented two-click import (Node Exporter Full, cAdvisor) rather than vendored JSON
  • Add docker/management-proxy/: TLS termination and name-based vhost routing (jenkins., grafana., uptime.) for every public UI on the management host, replacing the single-vhost jenkins/nginx.conf. Domain names are injected via the official nginx image's envsubst templating, not hardcoded
  • jenkins/docker-compose.yml: the nginx service moved out to docker/management-proxy/ - jenkins/ is Jenkins-only again

Roadmap Phase

v0.7.0 - Observability (PR B: central aggregation + UI + management-proxy refactor. PR A, already merged, instrumented every host with Node Exporter/cAdvisor. One v0.7.0 tag after both merge.)

Environment Impact

  • CI
  • Staging
  • Production
  • No environment impact yet (nothing deployed to a live host by this PR; AWS management host not provisioned when this was written, so no live Jenkins nginx setup to migrate)

Rollback Plan

Revert the merge commit; no live host runs any of this yet. jenkins/docker-compose.yml losing its nginx service has no live impact for the same reason.

Checklist

  • Commit messages follow Conventional Commits
  • CHANGELOG.md updated (under [Unreleased])
  • Scripts pass shellcheck (n/a - no shell scripts in this PR)
  • Docs updated (docker/observability/README.md, docker/management-proxy/README.md, jenkins/README.md, docker/README.md)

Test plan

  • docker compose config validates for docker/observability/ and docker/management-proxy/
  • Deploy on the management host in order (monitoring-agent -> jenkins -> observability -> management-proxy per docker/observability/README.md), confirm Grafana reachable at its vhost showing the provisioned Prometheus datasource
  • Confirm Prometheus's targets page shows node-exporter-management/cadvisor-management as up
  • Confirm Prometheus/Alertmanager are NOT reachable from outside observability_net
  • Confirm the envsubst-templated nginx.conf resolves the three domains correctly and does not mangle nginx's own $host/$remote_addr variables

Completes v0.7.0 (Observability): central metrics/alerting/dashboards
plus a shared TLS proxy for the management host.

- Add docker/observability/: Prometheus, Alertmanager, Grafana, Uptime
  Kuma. Prometheus/Alertmanager stay internal-only (neither has
  meaningful auth); Grafana/Uptime Kuma get a public URL through the
  new management proxy. Prometheus joins monitoring_agent_net to
  scrape node-exporter/cadvisor by container name - their
  127.0.0.1-published ports aren't reachable from inside another
  container, so same-host scraping goes over a shared Docker network
  instead
- Three default alert rules (host down, disk >85%, container missing)
  with a no-op Alertmanager receiver - a real notification channel
  needs deployment-specific credentials that don't belong in this repo
- Grafana's Prometheus datasource is provisioned as code; dashboards
  are a documented two-click import (Node Exporter Full, cAdvisor)
  rather than vendored JSON
- Add docker/management-proxy/: TLS termination and name-based vhost
  routing (jenkins.<domain>, grafana.<domain>, uptime.<domain>) for
  every public UI on the management host, replacing the single-vhost
  jenkins/nginx.conf. Domain names are injected via the official nginx
  image's envsubst templating, not hardcoded
- jenkins/docker-compose.yml: the nginx service moved out to
  docker/management-proxy/ - jenkins/ is Jenkins-only again

Image versions and healthcheck tooling (Grafana/Uptime Kuma's
available shell utilities, cAdvisor's base image, current registry
paths) verified via research rather than assumed.
@mcansem
mcansem merged commit d8bd57f into main Jul 21, 2026
2 checks passed
@mcansem
mcansem deleted the phase/v0.7.0-observability-stack branch July 22, 2026 10:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant