Active health checks with `health_port` mark all upstreams as unhealthy despite successful manual checks

[tnucera]

Environment

• Caddy version: 2.11.1 (Docker caddy:2.11.1-alpine)
• OS: Alpine Linux (container), host is Ubuntu/Debian
• Custom module: one custom Caddy middleware (document loader, unrelated to health checks)

Bug description

When using health_port to perform active health checks on a different port than the upstream's dial address, Caddy marks all upstreams as unhealthy (caddy_reverse_proxy_upstreams_healthy = 0) and returns 503 to all requests — even though the health endpoint is reachable and responds 200 OK when tested manually from inside the same container.

The upstreams never transition to healthy, regardless of how long Caddy runs.

Caddyfile (simplified, relevant parts)

The same reverse_proxy snippet is imported in multiple route blocks:

(proxy_config) {
	reverse_proxy 172.16.40.146:8761 172.16.40.147:8761 172.16.40.148:8761 172.16.40.157:8761 172.16.40.158:8761 {
		lb_policy header X-Document-ID

		health_uri /health
		health_port 8762
		health_interval 5s
		health_timeout 3s

		lb_try_duration 5s
		lb_try_interval 250ms

		fail_duration 30s
		unhealthy_status 500 502 503 504
		unhealthy_latency 30s

		transport http {
			dial_timeout 5s
			response_header_timeout 120s
		}
	}
}

:80

route /documents/*/layout {
	import proxy_config
}

route /documents/*/pages/* {
	import proxy_config
}

route /documents/*/file {
	import proxy_config
}

route {
	import proxy_config
}

Architecture

Each upstream host runs:

• Application server on port 8761 (the actual upstream for proxied traffic)
• Healthcheck sidecar on port 8762 — a lightweight Go HTTP server that returns:
  • 200 OK + {"status":"healthy"} when all co-located services are up
  • 503 + {"status":"unhealthy","failing":[...]} when any service is down

Port 8762 is exposed on the host via Docker port mapping (-p 8762:8762).

Steps to reproduce

1. Configure reverse_proxy with health_port 8762 and health_uri /health as shown above
2. Ensure the health endpoint on port 8762 returns 200 OK on all backends
3. Start Caddy
4. Wait >30 seconds (multiple health check cycles at 5s interval)
5. Check caddy_reverse_proxy_upstreams_healthy → all 0
6. Send any request through the proxy → 503 Service Unavailable

Evidence: health endpoint works from inside the Caddy container

/srv # for ip in 172.16.40.146 172.16.40.147 172.16.40.148 172.16.40.157 172.16.40.158; do
>     echo -n "$ip:8762 → "; wget -qO- http://$ip:8762/health 2>&1; echo
>   done
172.16.40.146:8762 → {"status":"healthy"}
172.16.40.147:8762 → {"status":"healthy"}
172.16.40.148:8762 → {"status":"healthy"}
172.16.40.157:8762 → {"status":"healthy"}
172.16.40.158:8762 → {"status":"healthy"}

Evidence: Caddy reports all upstreams as unhealthy

Prometheus metrics:

/srv # wget -qO- http://localhost:2019/metrics 2>&1 | grep upstreams_healthy
caddy_reverse_proxy_upstreams_healthy{upstream="172.16.40.146:8761"} 0
caddy_reverse_proxy_upstreams_healthy{upstream="172.16.40.147:8761"} 0
caddy_reverse_proxy_upstreams_healthy{upstream="172.16.40.148:8761"} 0
caddy_reverse_proxy_upstreams_healthy{upstream="172.16.40.157:8761"} 0
caddy_reverse_proxy_upstreams_healthy{upstream="172.16.40.158:8761"} 0

Admin API /reverse_proxy/upstreams:

[
  {"address":"172.16.40.147:8761","num_requests":0,"fails":0},
  {"address":"172.16.40.148:8761","num_requests":0,"fails":0},
  {"address":"172.16.40.157:8761","num_requests":0,"fails":0},
  {"address":"172.16.40.158:8761","num_requests":0,"fails":0},
  {"address":"172.16.40.146:8761","num_requests":0,"fails":0}
]

All upstreams show num_requests: 0 and fails: 0 (no traffic proxied, no passive failures). Combined with the Prometheus metrics above (upstreams_healthy = 0), this confirms the active health checker is the sole cause — it never marks any upstream as healthy.

Loaded JSON config (relevant excerpt)

The config loaded by Caddy confirms health_port is correctly parsed:

"health_checks": {
  "active": {
    "interval": 5000000000,
    "port": 8762,
    "timeout": 3000000000,
    "uri": "/health"
  },
  "passive": {
    "fail_duration": 30000000000,
    "unhealthy_latency": 30000000000,
    "unhealthy_status": [500, 502, 503, 504]
  }
},
"upstreams": [
  {"dial": "172.16.40.146:8761"},
  {"dial": "172.16.40.147:8761"},
  {"dial": "172.16.40.148:8761"},
  {"dial": "172.16.40.157:8761"},
  {"dial": "172.16.40.158:8761"}
]

This block appears 4 times in the config (once per route).

Expected behavior

After a few health check cycles (interval is 5s), Caddy should mark the upstreams as healthy (1) and proxy traffic normally.

Actual behavior

All upstreams remain at 0 indefinitely. Every request returns 503 Service Unavailable.

Analysis and hypotheses

1. The health endpoint is reachable — proven by wget from inside the container, same network namespace, same DNS resolution.

2. The config is correctly loaded — "port": 8762 and "uri": "/health" appear in the JSON config.

3. Passive health checks are not the cause — fails: 0 for all upstreams (no traffic has been proxied).

4. Possible interaction between multiple handlers: the snippet creates 4 independent reverse_proxy handlers (one per route), each with its own health checker, all targeting the same 5 backends. The Prometheus gauge caddy_reverse_proxy_upstreams_healthy is keyed only by upstream address — could multiple handlers writing to the same gauge interfere? Or could 4 concurrent health checkers sharing the same HTTPTransport connection pool cause issues?

5. Possible regression of Bug in active health checker in reverse_proxy using health_port? #3691: the original health_port bug (fixed in v2.2.0) was caused by the transport's DialContext reading the dial address from the request context instead of the URL. The fix was to "construct a fresh HTTP transport for health checks." However, the current code (v2.11.1) appears to use h.Transport (the handler's transport) for health check requests. For health checks, there's no DialInfo in the context, so the custom DialContext should fall through to using the URL's address — but this warrants verification.

Related issues

• Bug in active health checker in reverse_proxy using health_port? #3691 — health_port bug where transport connected to wrong port (fixed in v2.2.0)
• Reverse proxy startup health check behavior results in 503 errors #6410 — upstreams default to unhealthy before first check (open)
• Active health check not separated for same upstream #4341 — health check scoping per handler (fixed in reverseproxy: Dynamic upstreams (with support for SRV and A/AAAA lookups) #4470)
• admin: /reverse_proxy/upstreams endpoint should provide info on active health checks #6135 — /reverse_proxy/upstreams should expose active health check info

[francislavoie]

Duplicate of #7524