Skip to content

feat(daemon): caddy-l4 PROXY protocol — wrapping-aware lifecycle - #107

Merged
hsinatfootprintai merged 1 commit into
mainfrom
feat/daemon-l4-proxy-protocol
May 9, 2026
Merged

feat(daemon): caddy-l4 PROXY protocol — wrapping-aware lifecycle#107
hsinatfootprintai merged 1 commit into
mainfrom
feat/daemon-l4-proxy-protocol

Conversation

@hsinatfootprintai

@hsinatfootprintai hsinatfootprintai commented May 9, 2026

Copy link
Copy Markdown
Contributor

Summary

Closes the gRPC outage gap deferred from #106. Makes L4ProxyManager aware of the sandbox-tier-1-verified pattern B wrapping so:

  • ActivateL4 produces the wrapped shape directly when proxy-protocol is enabled
  • AddL4Route / RemoveL4Route / ListL4Routes operate on the inner subroute, leaving the wrapper intact
  • The previous attempt's regression (RouteSyncJob unwrapping the routes within seconds of daemon startup) can't happen

What pattern B is

caddy-l4 server :443
└─ outer route (no match)
   ├─ handle 0: layer4.handlers.proxy_protocol  (allow=[<sentinel-IP>/32], timeout=5s)
   └─ handle 1: layer4.handlers.subroute
      ├─ SNI route passthrough-a.example      → 203.0.113.1:50051   (raw TLS — gRPC backend)
      ├─ SNI route passthrough-b.example  → 203.0.113.2:50052   (raw TLS — gRPC backend)
      └─ catchall (no match)             → localhost:8443     (with proxy_protocol: "v2", srv0)

The proxy_protocol handler consumes the leading PROXY v2 bytes (lenient if absent) so SNI matching inside the subroute sees the underlying TLS ClientHello cleanly. The catchall re-emits PROXY to srv0 so srv0's listener_wrapper recovers the source.

How the lifecycle stays wrapping-aware

Manager method Behavior
SetProxyProtocolTrusted(cidrs) Records CIDRs on the manager. Validates non-empty, no wildcards.
EnableL4ProxyProtocol(cidrs) SetProxyProtocolTrusted + reshape if L4 already active.
ActivateL4 If proxy-protocol enabled, builds wrapped shape from the start; else flat (legacy).
getRoutes Atomic GET of full config; returns inner subroute's routes when wrapped, outer otherwise.
putRoutes Atomic POST /load with a config that preserves the wrapper structure — only the inner routes is replaced.
AddL4Route / RemoveL4Route / ListL4Routes Unchanged — call getRoutes/putRoutes, work transparently in both states.

Caddy admin's PATCH/replace semantics on the legacy /config/.../routes path can't accidentally wipe the outer wrapper because we never PATCH that path anymore — everything goes through atomic /load.

Tests

Unit (internal/app/l4_proxy_test.go)

  • _NotActive: EnableL4ProxyProtocol on inactive L4 records CIDRs without error.
  • _RejectsEmpty / _RejectsWildcard: same safety guards as the HTTP-side variant.
  • _ActivateL4_WrappedWhenEnabled: ActivateL4 with SetProxyProtocolTrusted set produces wrapped shape directly.
  • _Lifecycle_WrappingSurvivesRouteSyncJob — the regression test for the prod outage 3 bug. Simulates 3 RouteSyncJob cycles of AddL4Route + a RemoveL4Route + ListL4Routes, asserts wrapping invariant at every step.
  • _ReshapesActiveFlatServer: a flat-shape L4 server (e.g. left over from an older daemon) gets atomically reshaped without losing pre-existing SNI routes.

Tier-2 lifecycle driver (test/fixtures/tier2-l4-lifecycle/main.go)

Cross-compiled binary that exercises the same lifecycle against a real caddy-l4 2.11.2 binary on the sandbox. Verified on sandbox:

  • 6 add cycles, 1 remove, all preserve wrapping
  • Curl-driven e2e for all 4 deploy-state scenarios (PROXY×catchall/SNI, no-PROXY×catchall/SNI) — all pass
  • Final L4 config matches pattern B byte-for-byte

What ships in dual_server.go

In both EnsureServerConfig call sites, after EnableProxyProtocol(srv0), we now also call l4ProxyManager.EnableL4ProxyProtocol(config.ProxyProtocolTrusted) — same trusted CIDR list, same safety guards.

Pre-deploy plan (when ready)

  1. Merge this PR
  2. Rebuild + ship daemon binary
  3. Restart daemon (override already in place from feat(daemon): wire --proxy-protocol flags into Caddy startup (srv0 only) #106 deploy — same flags work)
  4. Verify Caddy admin shows wrapped L4 server (the lifecycle test gives us the exact expected shape)
  5. Restart sentinel with --proxy-protocol=true
  6. Verify wordpress XFF + gRPC route still works (= 415 to plain curl, valid TLS handshake)

🤖 Generated with Claude Code

Builds on #106. Closes the gRPC outage gap by making the L4ProxyManager
lifecycle aware of the sandbox-tier-1-verified pattern B wrapping.

Architecture:

When SetProxyProtocolTrusted (or EnableL4ProxyProtocol) records a non-empty
trustedSenderCIDRs on the manager:
  - ActivateL4 produces the wrapped shape directly: a single outer route
    whose handlers are (proxy_protocol, subroute), with the catchall inside
    the subroute tagged proxy_protocol: "v2" so caddy-l4 emits a PROXY v2
    header to srv0.
  - getRoutes uses the atomic full-config GET and returns the inner
    subroute's routes when wrapped, outer routes when flat.
  - putRoutes uses the atomic /load with a config that preserves the
    wrapping structure: only the inner subroute's `routes` is replaced;
    proxy_protocol handler stays intact.
  - AddL4Route / RemoveL4Route / ListL4Routes call getRoutes/putRoutes
    unchanged — they automatically operate on the inner subroute when
    wrapped.

This was the bug in attempt 3 (prod outage 3): EnableL4ProxyProtocol
wrapped the routes, then RouteSyncJob's AddL4Route used the old flat-
routes assumption and clobbered the wrapping within ~5s of daemon startup.

EnableL4ProxyProtocol(trustedCIDRs) is now: SetProxyProtocolTrusted +
reshape if L4 is already active. Calling it on a fresh daemon (L4 not
yet active) just records the CIDRs; ActivateL4 (called later by
RouteSyncJob) picks them up and produces the wrapped shape.

Tests:

  internal/app/l4_proxy_test.go:
  - _NotActive: EnableL4ProxyProtocol on inactive L4 records CIDRs.
  - _RejectsEmpty / _RejectsWildcard: same safety guards.
  - _ActivateL4_WrappedWhenEnabled: ActivateL4 with proxy-protocol set
    produces wrapped shape directly (single outer route, proxy_protocol
    + subroute, catchall tagged v2).
  - _Lifecycle_WrappingSurvivesRouteSyncJob: REGRESSION TEST for
    attempt 3. Runs 3 cycles of AddL4Route + asserts wrapping intact at
    every step, plus a final RemoveL4Route + ListL4Routes.
  - _ReshapesActiveFlatServer: an already-active L4 in legacy flat shape
    gets atomically re-shaped by EnableL4ProxyProtocol, with all
    pre-existing SNI routes preserved.

  test/fixtures/tier2-l4-lifecycle/main.go:
  - Cross-compiled driver that runs the same lifecycle against a real
    caddy-l4 binary. Validated on sandbox: wrapping survives 6 add/remove
    cycles + scenario 1-4 e2e (curl with/without PROXY × catchall/SNI).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@hsinatfootprintai
hsinatfootprintai merged commit f225948 into main May 9, 2026
6 of 7 checks passed
@hsinatfootprintai
hsinatfootprintai deleted the feat/daemon-l4-proxy-protocol branch May 9, 2026 13:03
pull Bot referenced this pull request in Spencerx/Containarium May 9, 2026
Architecture document covering the three-PR chain (#105, #106, #107) that
delivered real-client-IP propagation from the sentinel through caddy-l4 to
the daemon's HTTP server.

Sections:
- The problem statement (X-Forwarded-For: ::1 baseline before).
- The three-hop architecture diagram and what each layer does.
- Deploy state matrix (sentinel × daemon flag combinations) — explicitly
  flags the unsafe order (sentinel-on, daemon-off).
- Trust model: why two different allow CIDR scopes are needed, why
  wildcards are refused.
- Recommended rollout order + verification recipe (curl + nginx access log).
- Rollback paths for either side.
- Test inventory and pattern B reference config (with notes on the
  surprising parts that aren't obvious from caddy-l4 docs).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants