[Audit] S9 Safety: No Docker health checks in docker-compose services

[mvillmow]

Audit Finding

Severity: MAJOR
Section: 9. Safety & Reliability
Grade Impact: This finding contributes to the F grade for this section.

Summary

None of the five services in docker-compose.yml define Docker health checks. Without health checks, Docker (or Podman) cannot determine whether a service is actually healthy or just running. The depends_on directives only wait for container start, not service readiness, which can cause startup race conditions.

Evidence

• File: docker-compose.yml lines 6-89
• Observed: No healthcheck block in any of the five service definitions (prometheus, loki, promtail, grafana, argus-exporter)
• Expected: Each service should have a healthcheck block with appropriate test commands:
  • Prometheus: wget --spider http://localhost:9090/-/ready
  • Loki: wget --spider http://localhost:3100/ready
  • Grafana: wget --spider http://localhost:3000/api/health
  • Exporter: wget --spider http://localhost:9100/health

Principle Violation

POLA: Docker Compose users expect depends_on with condition: service_healthy to ensure proper startup ordering. Without health checks, services may start before their dependencies are ready, causing transient errors on first boot.

Recommendation

Add healthcheck blocks to each service in docker-compose.yml. Example for Prometheus:

healthcheck:
  test: ["CMD", "wget", "--spider", "-q", "http://localhost:9090/-/ready"]
  interval: 15s
  timeout: 5s
  retries: 3
  start_period: 10s

Then update depends_on to use condition: service_healthy.

Impact

Services may fail on initial scrape attempts after just start because dependencies are not yet ready. This leads to confusing error messages and requires manual restarts or waiting.

---

Filed by HomericIntelligence ecosystem audit (repo-analyze-strict methodology)
Audit date: 2026-03-22

[mvillmow]

Implementation Plan

Promtail has http_listen_port: 9080 — its /ready endpoint is available at http://localhost:9080/ready.

---

3. Files to Create

None. This change is contained entirely within docker-compose.yml.

---

4. Files to Modify

docker-compose.yml — the only file that changes

Service	Health Endpoint	Port	Notes
prometheus	/-/ready	9090	Standard Prom readiness
loki	/ready	3100	Standard Loki readiness
promtail	/ready	9080	Server block already configured
grafana	/api/health	3000	Add GF_ANALYTICS_* env vars
argus-exporter	/health	9100	Assumes exporter.py exposes this

Verify before implementing: Confirm exporter.py has a /health route.

depends_on upgrades:

• promtail: loki → condition: service_healthy
• grafana: prometheus + loki → both condition: service_healthy

Healthcheck parameters (uniform across all services):

interval: 15s
timeout: 5s
retries: 3
start_period: 15s

---

5. Implementation Order

1. Verify exporter.py has a /health endpoint — read the file before writing any healthcheck for argus-exporter
2. Add healthcheck to prometheus (no dependencies; safe to do first)
3. Add healthcheck to loki (no dependencies)
4. Add healthcheck to promtail + upgrade its depends_on: loki to condition: service_healthy
5. Add healthcheck to grafana + add GF_ANALYTICS_* env vars + upgrade depends_on to condition: service_healthy for both prometheus and loki
6. Add healthcheck to argus-exporter (standalone, no depends_on)
7. Run just start to verify the stack comes up cleanly

---

6. Verification

# 1. Bring up the stack and watch health status
just start
docker compose ps   # all services should show "(healthy)" status

# 2. Confirm healthcheck details per service
docker inspect argus-prometheus | grep -A 20 '"Health"'
docker inspect argus-loki       | grep -A 20 '"Health"'
docker inspect argus-promtail   | grep -A 20 '"Health"'
docker inspect argus-grafana    | grep -A 20 '"Health"'
docker inspect argus-exporter   | grep -A 20 '"Health"'

# 3. Verify depends_on ordering worked (no scrape errors at t=0)
just test-scrape   # all targets should show up=1 immediately
just logs prometheus | head -50  # no "connection refused" errors

Expected outcome: All five containers reach (healthy) state before just test-scrape is run. No transient up=0 readings in Prometheus for agamemnon, nestor, nats, or nomad jobs immediately after startup.

---

7. Skills Used

Invoked during planning:

• None directly invoked (plan derived from issue content, file reads, and prior learnings)

Team knowledge base skills referenced (from Prior Learnings):

• e2e-homeric-compose-cpp-pipeline — healthcheck patterns for Prometheus, Grafana, Loki, Promtail compose stacks; source of the CMD-SHELL vs CMD format distinction
• podman-scratch-image-healthcheck-workaround — confirmed none of the five images are scratch/distroless (so depends_on: condition: service_healthy will not block indefinitely); documented Grafana startup hang fix via GF_ANALYTICS_* env vars

---

Generated by Claude Code Planner