reverseproxy: isolate active health-check state per distinct check config - #7916
Open
SillyZir wants to merge 2 commits into
Open
reverseproxy: isolate active health-check state per distinct check config#7916SillyZir wants to merge 2 commits into
SillyZir wants to merge 2 commits into
Conversation
…nfig Multiple reverse_proxy handlers configured with different active health checks (health_uri, health_headers, ...) against the same upstream dial address currently share a single Host in the global pool, so one handler's failing probes mark the address unhealthy for every other handler. Key the pool by dial address plus a stable fingerprint of the active health-check config, so distinct checks get independent health state. The fingerprint is strictly internal to pool identity: the Prometheus upstreams_healthy label and the /reverse_proxy/upstreams admin endpoint continue to report the plain dial address, unchanged. Dynamic upstreams are intentionally out of scope here: they resolve through a separate per-lookup path (dynamicHosts) and collapsing there has different lifetime semantics; noted for a follow-up. Fixes caddyserver#7870
Satisfies the modernize linter; behaviour is unchanged, since Cut returns the whole string when the separator is absent.
steadytao
requested changes
Aug 10, 2026
steadytao
left a comment
steadytao
left a comment
Member
There was a problem hiding this comment.
Each handler still writes caddy_reverse_proxy_upstreams_healthy using only upstream.Dial. Distinct health-check states therefore race to update the same Prometheus series so the reported value depends on which updater ran last. The metric identity needs to match the newly separated health-state identity + comment
| } | ||
| // pool keys may carry an internal health-check fingerprint | ||
| // (see hostKeySuffix); report only the plain dial address | ||
| address := hostKeyAddress(poolKey) |
Member
There was a problem hiding this comment.
Existing consumers that index by address will silently discard all but one. This needs defined aggregation semantics or a public discriminator.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #7870.
Multiple
reverse_proxyhandlers configured with different active health checks (health_uri,health_headers, ...) against the same upstream dial address currently share a singleHostin the global pool, so one handler's failing probes mark the address unhealthy for every other handler. This keys the pool by dial address plus a stable fingerprint of the active health-check config, so distinct checks get independent health state.The fingerprint is strictly internal to pool identity:
upstreams_healthylabel is untouched (still the plainDialaddress);/reverse_proxy/upstreamsadmin endpoint strips the fingerprint at emission (hostKeyAddress), so API output reports plain dial addresses as before.Scope note: dynamic upstreams are intentionally not covered here — they resolve through the separate per-lookup
dynamicHostspath with different lifetime semantics, so the same collapse there is left for a follow-up rather than half-fixed in this change.The regression test (
TestActiveHealthChecksSameAddressDifferentChecksAreIndependent) spins up two handlers with differenthealth_uriagainst one address and asserts vhost B's health state stays independent while vhost A's probes fail; it fails on master and passes with this change.Files:
modules/caddyhttp/reverseproxy/{healthchecks,hosts,reverseproxy,admin}.go,active_health_test.goAssistance Disclosure
This patch was developed with AI assistance. It was reviewed and tested before submission (affected package tests pass), and is submitted from a human-owned account that takes responsibility for the change.