feat(daemon): wire --proxy-protocol flags into Caddy startup (srv0 only) - #106
Merged
Conversation
Adds the rollout-half of the PROXY-protocol work merged in #105: when --proxy-protocol is set, the daemon calls ProxyManager.EnableProxyProtocol right after EnsureServerConfig, installing the [proxy_protocol, tls] listener_wrappers + trusted_proxies on the running Caddy. Flags: --proxy-protocol Off by default. Off behavior is unchanged. --proxy-protocol-trusted CIDR allow list for PROXY senders. Defaults to 127.0.0.0/8 (tunnel/local). Wildcards (0.0.0.0/0, ::/0) are rejected by EnableProxyProtocol to prevent IP spoofing. Both EnsureServerConfig call sites in dual_server.go (the app-hosting branch and the route-store recovery branch) get the same wiring, so either bring-up path picks up the wrapper. Pair with `containarium sentinel --proxy-protocol` to complete the chain. Recommended deploy order: daemon first (wrapper degrades gracefully for non-PROXY connections from allowed CIDRs), verify HTTPS still works, then flip the sentinel flag. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Production deploys run a caddy-l4 server in front of srv0 — :443 is owned by caddy-l4, which TLS-passthrough-routes by SNI, with a catch-all that proxies to srv0 at localhost:8443. With only PR #105's sentinel side and PR #106's srv0 wrapper, caddy-l4 would either fail SNI matching on the leading PROXY bytes or strip them at the loopback hop — either way the real client IP never reaches srv0 and X-Forwarded-For ends up as ::1 at the user container (verified the ::1 baseline against wordpress.kafeido.app). Adds L4ProxyManager.EnableL4ProxyProtocol(trustedCIDRs): - Installs a `proxy_protocol` listener_wrapper on the tls_passthrough server so the leading PROXY v2 bytes are stripped + the parsed source becomes conn.RemoteAddr before SNI matching runs. - Tags every proxy handler with `proxy_protocol: "v2"` so caddy-l4 re-emits a PROXY header to its upstream — both the localhost:8443 catch-all (where srv0's wrapper from PR #106 picks it up) and any SNI routes (gRPC LXCs) that happen to speak PROXY. - Idempotent no-op when L4 isn't active — it'll be applied next time L4 activates if the daemon was started with --proxy-protocol. - Same empty/wildcard CIDR validation as the srv0 EnableProxyProtocol. Wires the call into both EnsureServerConfig sites in dual_server.go right after the srv0 PROXY-protocol patch, sharing the same trusted CIDR list. Tests: - in-process fake Caddy admin (httptest) with a mini /load endpoint proves the patched config has the right shape: listener_wrappers at the server level + proxy_protocol on every proxy handler. - inactive-L4 path is a no-op. - empty/wildcard CIDR rejection. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Author
|
Extended this PR to also patch the While verifying the deploy plan against prod ( Today's baseline at the wordpress nginx is New piece:
Wired into both Recommended ( |
…load PROD INCIDENT: deployed PR #106 to backend, EnableProxyProtocol PATCHed /config/apps/http/servers/srv0 with a body containing only listener_wrappers and trusted_proxies. Caddy admin's PATCH semantics REPLACE the resource at the path (they don't merge fields), so srv0 lost listen, routes, automatic_https, and tls_connection_policies. RouteSyncJob then failed every sync with "json: cannot unmarshal object into Go struct field Server.servers.routes of type caddyhttp.RouteList" and HTTPS broke for every subdomain that flowed through srv0 (incl. wordpress.kafeido.app). Mitigation was a manual rollback (DELETE srv0 + restore old binary + systemd-reload). Total downtime ~5 min. Fix: mirror the L4 manager's atomic-/load pattern — GET full config, set the two new fields on the in-memory map (preserving everything else), POST /load. Adds local getFullConfig/loadConfig helpers on ProxyManager that match the L4ProxyManager pair (extract-to-shared is a follow-up cleanup). Regression test: TestProxyManager_EnableProxyProtocol_PreservesOtherFields seeds the fake Caddy with srv0 having listen + routes + automatic_https, runs EnableProxyProtocol, then asserts every pre-existing field is still present after the call. The previous test only verified the request body shape (which looked correct); the new test exercises the actual semantics. The old test (TestProxyManager_EnableProxyProtocol with the simple httptest stub) was deleted — its assertions about "method = PATCH" and "path = /config/apps/http/servers/srv0" are exactly the wrong-by-design behavior. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Contributor
Author
|
Prod incident + fix in commit d7b66b0. Deployed this PR to backend earlier; the daemon came up but Fix: mirror the New regression test Re-deploy plan unchanged. Will re-attempt after merge — same procedure, same flags. |
…ers)
Discovered on prod: caddy-l4 has no server-level listener_wrappers field —
attempting to set it returns 400 with "unknown field listener_wrappers".
EnableL4ProxyProtocol's previous implementation hit this and the L4 patch
silently no-op'd (just a warning log), leaving caddy-l4 PROXY-unaware.
Replace the listener_wrappers approach with caddy-l4's canonical pattern:
- Wrap existing routes in a subroute under a top-level route whose match
list contains the proxy_protocol matcher. The matcher consumes the
PROXY header during the match phase, so the subroute's SNI matchers
see the underlying TLS bytes.
- Tag every proxy handler in the wrapped subroute with proxy_protocol:
"v2" so caddy-l4 re-emits a PROXY header to upstream (srv0 / gRPC LXC)
carrying the parsed real client IP.
- Add a fallback top-level route (no match clause) that mirrors the
original routes verbatim (no proxy_protocol emission). This prevents a
deploy-gap outage: when the daemon is flipped but the sentinel hasn't
been yet, raw-TLS connections still flow through the fallback. After
the sentinel flips, the fallback becomes dead code.
- Idempotent: detect "already wrapped" by checking the first route's
match list for the proxy_protocol matcher; skip if present.
Tests:
- _WrapsRoutes asserts the new outer-shape: 2 outer routes, first with
proxy_protocol matcher + subroute (proxy handlers tagged v2), second
is the no-match fallback (handlers MUST NOT have proxy_protocol).
- _Idempotent asserts a second call doesn't double-wrap.
- Old _PatchesActive test deleted — its assertions about server-level
listener_wrappers were exactly the wrong-by-design behavior.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…er_wrappers)" This reverts commit ea928d6.
…he L4 hop" This reverts commit 9d19811.
The two L4 commits (9d19811, ea928d6) carried these helpers; reverting them broke the srv0 fix in proxy.go and its test. Re-add toAnySlice as a private helper in proxy.go (one caller now) and inline newFakeCaddy in proxy_test.go so the regression test still runs. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
hsinatfootprintai
changed the title
…rified)
After two prod incidents trying various caddy-l4 PROXY configs, set up a
real Caddy 2.11.2 + caddy-l4 sandbox stand and tested 4 patterns against
the actual binary. Pattern B is the only one that works in all 4
deploy-state scenarios.
Pattern B (verified-good shape):
L4 server has ONE outer route whose handlers are:
1. layer4.handlers.proxy_protocol — consumes PROXY v2 header from
trusted sender CIDRs, lenient on missing PROXY (passes through
unchanged, so deploy-gap traffic still flows).
2. layer4.handlers.subroute — does SNI matching on now-clean TLS
bytes. Only the catchall inside the subroute is tagged with
proxy_protocol: "v2" so caddy-l4 emits a PROXY header to srv0
(which has its own listener_wrapper from EnableProxyProtocol).
SNI passthrough routes are left untagged because gRPC backends
don't speak PROXY and just want raw TLS.
Verified scenarios (sandbox tier 1, real Caddy):
1. PROXY + catchall → wordpress backend sees real client IP via XFF.
2. PROXY + SNI route → gRPC backend gets clean TLS bytes (PROXY
consumed by handler, SNI matching succeeds post-strip).
3. no-PROXY + catchall → flows through; XFF is the L4 IP, harmless.
4. no-PROXY + SNI route → flows through unchanged; legacy behavior.
Wrong patterns ruled out by tier 1 (saved as the file's commentary):
- listener_wrappers at L4 server level → caddy-l4 has no such field.
- proxy_protocol MATCHER → silently dropped connections.
- AND-ed matchers in a single match clause → broke catchall.
- allow field on the matcher → "unknown field allow".
Tier 2 (real daemon binary against sandbox Caddy):
cross-compiled containarium daemon, called EnableL4ProxyProtocol against
the sandbox Caddy admin, read back the resulting config — confirmed
matches pattern B byte-for-byte. test/fixtures/tier2-l4-driver/main.go
is the harness that runs that loop.
Code refactor (responding to "use struct instead of raw map"):
- New typed structs in caddy_types.go: CaddyL4ProxyProtocolHandler,
CaddyL4SubrouteHandler, CaddyL4ProxyHandler, CaddyL4WrappedOuterRoute.
- EnableL4ProxyProtocol now constructs the new outer route with the
typed structs (only existing routes stay as []interface{} so unknown
fields aren't dropped).
- EnableProxyProtocol on the HTTP side similarly uses CaddyListenerWrapper
and CaddyTrustedProxies structs instead of literal maps.
- Dropped the toAnySlice helper (no callers).
Wire-up: dual_server.go calls EnableL4ProxyProtocol after EnableProxyProtocol
in both bring-up paths, sharing the same trusted CIDR list (extra entries
on either side are harmless because each side's allow only matches its
actual senders).
Tests:
- _NotActive: no-op when L4 isn't in the running config.
- _WrapsRoutes: asserts the full pattern B shape (single outer route,
proxy_protocol+subroute handlers, catchall tagged v2, SNI route
NOT tagged).
- _Idempotent: a second invocation against an already-wrapped server
is a no-op (no double-nesting).
- _RejectsEmpty / _RejectsWildcard: same safety guards as
EnableProxyProtocol.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ndbox-verified)" This reverts commit b78bec6.
hsinatfootprintai
added a commit
that referenced
this pull request
May 9, 2026
Builds on #106. Closes the gRPC outage gap by making the L4ProxyManager lifecycle aware of the sandbox-tier-1-verified pattern B wrapping. Architecture: When SetProxyProtocolTrusted (or EnableL4ProxyProtocol) records a non-empty trustedSenderCIDRs on the manager: - ActivateL4 produces the wrapped shape directly: a single outer route whose handlers are (proxy_protocol, subroute), with the catchall inside the subroute tagged proxy_protocol: "v2" so caddy-l4 emits a PROXY v2 header to srv0. - getRoutes uses the atomic full-config GET and returns the inner subroute's routes when wrapped, outer routes when flat. - putRoutes uses the atomic /load with a config that preserves the wrapping structure: only the inner subroute's `routes` is replaced; proxy_protocol handler stays intact. - AddL4Route / RemoveL4Route / ListL4Routes call getRoutes/putRoutes unchanged — they automatically operate on the inner subroute when wrapped. This was the bug in attempt 3 (prod outage 3): EnableL4ProxyProtocol wrapped the routes, then RouteSyncJob's AddL4Route used the old flat- routes assumption and clobbered the wrapping within ~5s of daemon startup. EnableL4ProxyProtocol(trustedCIDRs) is now: SetProxyProtocolTrusted + reshape if L4 is already active. Calling it on a fresh daemon (L4 not yet active) just records the CIDRs; ActivateL4 (called later by RouteSyncJob) picks them up and produces the wrapped shape. Tests: internal/app/l4_proxy_test.go: - _NotActive: EnableL4ProxyProtocol on inactive L4 records CIDRs. - _RejectsEmpty / _RejectsWildcard: same safety guards. - _ActivateL4_WrappedWhenEnabled: ActivateL4 with proxy-protocol set produces wrapped shape directly (single outer route, proxy_protocol + subroute, catchall tagged v2). - _Lifecycle_WrappingSurvivesRouteSyncJob: REGRESSION TEST for attempt 3. Runs 3 cycles of AddL4Route + asserts wrapping intact at every step, plus a final RemoveL4Route + ListL4Routes. - _ReshapesActiveFlatServer: an already-active L4 in legacy flat shape gets atomically re-shaped by EnableL4ProxyProtocol, with all pre-existing SNI routes preserved. test/fixtures/tier2-l4-lifecycle/main.go: - Cross-compiled driver that runs the same lifecycle against a real caddy-l4 binary. Validated on sandbox: wrapping survives 6 add/remove cycles + scenario 1-4 e2e (curl with/without PROXY × catchall/SNI). Co-authored-by: hsinhoyeh <yhh92u@gmail.com> Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 9, 2026
pull Bot
referenced
this pull request
in Spencerx/Containarium
May 9, 2026
Architecture document covering the three-PR chain (#105, #106, #107) that delivered real-client-IP propagation from the sentinel through caddy-l4 to the daemon's HTTP server. Sections: - The problem statement (X-Forwarded-For: ::1 baseline before). - The three-hop architecture diagram and what each layer does. - Deploy state matrix (sentinel × daemon flag combinations) — explicitly flags the unsafe order (sentinel-on, daemon-off). - Trust model: why two different allow CIDR scopes are needed, why wildcards are refused. - Recommended rollout order + verification recipe (curl + nginx access log). - Rollback paths for either side. - Test inventory and pattern B reference config (with notes on the surprising parts that aren't obvious from caddy-l4 docs). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Wires the daemon-side half of the PROXY-protocol work merged in #105, scoped to the HTTP server (srv0) only. The L4 layer (caddy-l4) is deliberately NOT touched in this PR — see "Scope reduction" below for why.
When
--proxy-protocolis set, the daemon callsProxyManager.EnableProxyProtocol(trustedCIDRs)afterEnsureServerConfig, atomically GET-modify-loading the Caddy config so srv0 gets a[proxy_protocol, tls]listener_wrappers chain andtrusted_proxiesstatic range list, while preserving listen/routes/automatic_https/etc.New flags
--proxy-protocolfalse--proxy-protocol-trusted127.0.0.0/8EnableProxyProtocol.Scope reduction (why no L4 in this PR)
Originally also extended caddy-l4 to be PROXY-aware (commits
9d19811+ea928d6, now reverted). Two prod incidents during testing showed I don't have enough hands-on caddy-l4 expertise to one-shot the L4 config:d7b66b0fixes it): naivePATCH /config/.../srv0REPLACES the resource at the path; my body containing onlylistener_wrappers+trusted_proxiesclobbered listen/routes. Fix: atomicgetFullConfig+loadConfigpattern. Regression test added.listener_wrappersto caddy-l4's server — caddy-l4 has no such field at the server level. Switched to aproxy_protocolmatcher + subroute restructure, but caddy-l4 silently dropped both the matcher andsubroute.routes, leaving an empty wrapper that broke routing.Decision: ship srv0 alone now, redo the L4 work in a follow-up PR after iterating against a local caddy-l4 sandbox to nail down the exact JSON shape with real Caddy validation in the loop.
What works after this PR alone (post sentinel
--proxy-protocol):X-Forwarded-For(caddy-l4'stlsSNI matcher fails-fast on PROXY bytes; the catch-all forwards bytes including the PROXY header to srv0; srv0's wrapper parses).What still won't work without the L4 follow-up:
passthrough-a.example,passthrough-b.example): break under PROXY because their SNI matchers fail on PROXY-prefix bytes and the catch-all has no gRPC route. Don't flip the sentinel--proxy-protocolflag in prod until the L4 follow-up is done if you need those routes working.What's in this PR
internal/cmd/daemon.go--proxy-protocoland--proxy-protocol-trustedflags; threaded intoDualServerConfiginternal/server/dual_server.goDualServerConfig.ProxyProtocol+.ProxyProtocolTrustedfields. BothEnsureServerConfigcall sites callEnableProxyProtocolwhen the flag is on.internal/app/caddy_types.goCaddyListenerWrapper,CaddyTrustedProxiestypes;CaddyServerConfigextended (HTTP types only — L4 types unchanged from main)internal/app/proxy.goEnableProxyProtocol(trustedCIDRs)rewritten to use atomicgetFullConfig→ mutate srv0 in place →POST /load. Local helpersgetFullConfig,loadConfig,toAnySlice.internal/app/proxy_test.goTestProxyManager_EnableProxyProtocol_PreservesOtherFields— seeds srv0 withlisten+routes+automatic_https, callsEnableProxyProtocol, asserts every pre-existing field still present. Plus the empty/wildcard CIDR rejection tests. InlinednewFakeCaddytest helper that matches real Caddy admin semantics (/loadis the only mutation path).Test plan
go build ./...cleango test ./internal/app/... ./internal/sentinel/...greencontainarium daemon --help | grep proxy-protocolshows the new flagslisten/routespreserved; HTTPS to existing subdomains still works (verified). Sentinel NOT flipped — wordpress.kafeido.app still shows baselineXFF: "::1".Follow-up (separate PR)
🤖 Generated with Claude Code