feat(daemon): caddy-l4 PROXY protocol — wrapping-aware lifecycle - #107
Merged
Conversation
Builds on #106. Closes the gRPC outage gap by making the L4ProxyManager lifecycle aware of the sandbox-tier-1-verified pattern B wrapping. Architecture: When SetProxyProtocolTrusted (or EnableL4ProxyProtocol) records a non-empty trustedSenderCIDRs on the manager: - ActivateL4 produces the wrapped shape directly: a single outer route whose handlers are (proxy_protocol, subroute), with the catchall inside the subroute tagged proxy_protocol: "v2" so caddy-l4 emits a PROXY v2 header to srv0. - getRoutes uses the atomic full-config GET and returns the inner subroute's routes when wrapped, outer routes when flat. - putRoutes uses the atomic /load with a config that preserves the wrapping structure: only the inner subroute's `routes` is replaced; proxy_protocol handler stays intact. - AddL4Route / RemoveL4Route / ListL4Routes call getRoutes/putRoutes unchanged — they automatically operate on the inner subroute when wrapped. This was the bug in attempt 3 (prod outage 3): EnableL4ProxyProtocol wrapped the routes, then RouteSyncJob's AddL4Route used the old flat- routes assumption and clobbered the wrapping within ~5s of daemon startup. EnableL4ProxyProtocol(trustedCIDRs) is now: SetProxyProtocolTrusted + reshape if L4 is already active. Calling it on a fresh daemon (L4 not yet active) just records the CIDRs; ActivateL4 (called later by RouteSyncJob) picks them up and produces the wrapped shape. Tests: internal/app/l4_proxy_test.go: - _NotActive: EnableL4ProxyProtocol on inactive L4 records CIDRs. - _RejectsEmpty / _RejectsWildcard: same safety guards. - _ActivateL4_WrappedWhenEnabled: ActivateL4 with proxy-protocol set produces wrapped shape directly (single outer route, proxy_protocol + subroute, catchall tagged v2). - _Lifecycle_WrappingSurvivesRouteSyncJob: REGRESSION TEST for attempt 3. Runs 3 cycles of AddL4Route + asserts wrapping intact at every step, plus a final RemoveL4Route + ListL4Routes. - _ReshapesActiveFlatServer: an already-active L4 in legacy flat shape gets atomically re-shaped by EnableL4ProxyProtocol, with all pre-existing SNI routes preserved. test/fixtures/tier2-l4-lifecycle/main.go: - Cross-compiled driver that runs the same lifecycle against a real caddy-l4 binary. Validated on sandbox: wrapping survives 6 add/remove cycles + scenario 1-4 e2e (curl with/without PROXY × catchall/SNI). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This was referenced May 9, 2026
pull Bot
referenced
this pull request
in Spencerx/Containarium
May 9, 2026
Architecture document covering the three-PR chain (#105, #106, #107) that delivered real-client-IP propagation from the sentinel through caddy-l4 to the daemon's HTTP server. Sections: - The problem statement (X-Forwarded-For: ::1 baseline before). - The three-hop architecture diagram and what each layer does. - Deploy state matrix (sentinel × daemon flag combinations) — explicitly flags the unsafe order (sentinel-on, daemon-off). - Trust model: why two different allow CIDR scopes are needed, why wildcards are refused. - Recommended rollout order + verification recipe (curl + nginx access log). - Rollback paths for either side. - Test inventory and pattern B reference config (with notes on the surprising parts that aren't obvious from caddy-l4 docs). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes the gRPC outage gap deferred from #106. Makes
L4ProxyManageraware of the sandbox-tier-1-verified pattern B wrapping so:ActivateL4produces the wrapped shape directly when proxy-protocol is enabledAddL4Route/RemoveL4Route/ListL4Routesoperate on the inner subroute, leaving the wrapper intactWhat pattern B is
The
proxy_protocolhandler consumes the leading PROXY v2 bytes (lenient if absent) so SNI matching inside the subroute sees the underlying TLS ClientHello cleanly. The catchall re-emits PROXY to srv0 so srv0's listener_wrapper recovers the source.How the lifecycle stays wrapping-aware
SetProxyProtocolTrusted(cidrs)EnableL4ProxyProtocol(cidrs)SetProxyProtocolTrusted+ reshape if L4 already active.ActivateL4getRoutesputRoutes/loadwith a config that preserves the wrapper structure — only the innerroutesis replaced.AddL4Route/RemoveL4Route/ListL4RoutesgetRoutes/putRoutes, work transparently in both states.Caddy admin's PATCH/replace semantics on the legacy
/config/.../routespath can't accidentally wipe the outer wrapper because we never PATCH that path anymore — everything goes through atomic/load.Tests
Unit (
internal/app/l4_proxy_test.go)_NotActive:EnableL4ProxyProtocolon inactive L4 records CIDRs without error._RejectsEmpty/_RejectsWildcard: same safety guards as the HTTP-side variant._ActivateL4_WrappedWhenEnabled:ActivateL4withSetProxyProtocolTrustedset produces wrapped shape directly._Lifecycle_WrappingSurvivesRouteSyncJob— the regression test for the prod outage 3 bug. Simulates 3 RouteSyncJob cycles ofAddL4Route+ aRemoveL4Route+ListL4Routes, asserts wrapping invariant at every step._ReshapesActiveFlatServer: a flat-shape L4 server (e.g. left over from an older daemon) gets atomically reshaped without losing pre-existing SNI routes.Tier-2 lifecycle driver (
test/fixtures/tier2-l4-lifecycle/main.go)Cross-compiled binary that exercises the same lifecycle against a real
caddy-l42.11.2 binary on the sandbox. Verified on sandbox:What ships in
dual_server.goIn both
EnsureServerConfigcall sites, afterEnableProxyProtocol(srv0), we now also calll4ProxyManager.EnableL4ProxyProtocol(config.ProxyProtocolTrusted)— same trusted CIDR list, same safety guards.Pre-deploy plan (when ready)
--proxy-protocol=true🤖 Generated with Claude Code