Skip to content

feat: l7 proxy dispatches mtls upstreams to client_dial - #444

Merged
kacy merged 3 commits into
mainfrom
feat/mtls-proxy-outbound
Jun 23, 2026
Merged

feat: l7 proxy dispatches mtls upstreams to client_dial#444
kacy merged 3 commits into
mainfrom
feat/mtls-proxy-outbound

Conversation

@kacy

@kacy kacy commented Jun 22, 2026

Copy link
Copy Markdown
Owner

summary

PR-5 final wiring (after #440, #441, #442, #443). teaches the L7 reverse
proxy's outbound forwardSingleAttempt to dispatch on
upstream.peer_mode and use client_dial.dial (#442) when mTLS is in
play. with this PR, the data-plane code path is complete — once a
service's tls.peer reaches the service-registry state (the one
remaining cluster-DB schema migration), inbound and outbound legs
both run mTLS end-to-end.

what's in here

split into two commits for review:

1. carry peer_mode through the spec-side types

  • Upstream gains a peer_mode field (default .off)
  • ServiceDefinition, ServiceState, ServiceSnapshot gain
    peer_mode and the apply/snapshot paths propagate it
  • resolveUpstreamWithPolicy copies peer_mode from the service
    snapshot onto every candidate Upstream and onto the returned one

2. dispatch + mTLS attempt in the reverse proxy

  • forwardSingleAttempt early-branches on upstream.peer_mode != .off
    into a new forwardSingleAttemptMtls:
    • loads the cluster CA from raft state (store.getClusterCa)
    • dials via client_dial.dial(ca_cert_pem=...)
    • writes the request and drains the response through the session
    • the connection is never pooled (mTLS sessions carry encryption
      state — sharing one across requests would corrupt the stream)
  • policy when the cluster CA isn't seeded:
    • .require ⇒ returns error.ClusterCaMissing
    • .warn ⇒ logs and falls back to plaintext via the new
      forwardPlainAttempt helper (an exact copy of the legacy leg
      of forwardSingleAttempt, so service traffic keeps flowing
      while the bootstrap thread catches up)
  • new readResponseFromSession: session-aware drain that reads
    until error.PeerClosed (the session's orderly EOF) or max_bytes

design notes

  • no session pool: the L7 connection pool keys on (endpoint_id, address, port) and stores raw fds. mixing fds and TLS sessions
    under one key would yield catastrophic checkouts (a session's
    encryption counters can't be picked up by a fresh handshake at
    the same address). a session pool would need a separate per-target
    map and a "is this session still healthy" check; left as a follow-up
    once we see real throughput numbers.
  • policy at the boundary: .warn falling back to plaintext is the
    spec's intent — it's the migration mode that proves the rest of the
    stack works before flipping a service to .require. when the CA is
    present, both .warn and .require run the same handshake.
  • mirror + http/2 paths: unchanged. they call
    socket_helpers.connectToUpstream directly. those paths get the
    same treatment in a follow-up once we have HTTP/1.1 mTLS in
    production.

tests

three new tests on readResponseFromSession against a duck-typed
FakeSession (same shape client_session.ClientSession exposes):

  1. concatenates chunks until peer closes
  2. returns ResponseTooLarge when max_bytes is exceeded
  3. returns an empty buffer on immediate close

the live mTLS handshake itself is already exhaustively covered by:

what's deferred (and why)

  • cluster-state-DB column for peer_mode — the bridge from the
    manifest loader → service store row → service registry. without
    it, Upstream.peer_mode stays .off in production. small dedicated
    migration: add a column, update services_types.zig + the raft
    INSERT/UPDATE SQL builders, copy through in
    serviceDefinitionFromRecord. one-shot schema PR; out of scope for
    this one.
  • session pool — see design notes; revisit with real throughput data.
  • mirror + HTTP/2 dispatch — same pattern, follow-up once HTTP/1.1
    mTLS has burn-in.

verification

  • full unit suite green (2234 passed, 2 skipped, 0 failed)
  • tools/panic_audit.sh clean
  • zig fmt --check clean

still no auto-merge — security-sensitive data-plane.

kacy added 3 commits June 22, 2026 21:56
adds tls.peer to the spec-side types so the L7 proxy can read the
service's mtls posture at dial time:

- Upstream gains peer_mode (defaults .off)
- ServiceDefinition and ServiceState gain peer_mode (defaults .off)
- ServiceSnapshot exposes peer_mode read-only
- assignCompatProxyFields copies peer_mode from def to state
- snapshotService propagates peer_mode
- resolveUpstreamWithPolicy copies peer_mode onto each Upstream
  candidate and onto the selected one returned to the caller

the cluster-state-DB column to persist peer_mode end-to-end (so the
manifest's tls.peer reaches resolveUpstream in production) is a small
follow-up schema migration; this PR's purpose is the in-memory carry
so forwardSingleAttempt can dispatch on it.
upstream.peer_mode != .off now branches into a new
forwardSingleAttemptMtls:
- loads the cluster CA via store.getClusterCa
- calls client_dial.dial with the CA pem + sni
- writes the request and drains the response through the session,
  no pooling (mtls sessions hold encryption state and can't share a
  bare-fd pool key)

policy:
- .require + missing cluster CA → ClusterCaMissing
- .warn + missing cluster CA → log + downgrade to the plaintext
  dial+pool path (kept as forwardPlainAttempt, exact copy of the
  legacy leg) so service-to-service traffic keeps flowing while the
  ca_bootstrap thread catches up

readResponseFromSession is the session-aware equivalent of the
existing bare-fd readResponse: drains chunks until PeerClosed (the
session's orderly EOF) or max_bytes, returns the bytes.

three tests on a duck-typed FakeSession cover the read loop's
happy path, max-bytes rejection, and immediate-close fallthrough.
the live tls handshake itself is already covered end-to-end by
the socketpair tests in #438/#439/#442.
closes the manifest → registry → upstream loop. previously peer_mode
was added to all the in-memory types but production code paths could
not set it because the cluster state DB had no column. with this
commit:

- services.peer_mode TEXT column (default 'off'); ALTER TABLE
  migration for existing databases (no NOT NULL so the migration is
  forward-safe).
- ServiceRecord.peer_mode (?[]const u8 — null means 'off', keeps
  every existing literal compiling).
- services_core.createInDb inserts peer_mode; syncConfig takes a
  peer_mode arg and updates the column alongside lb_policy.
- syncServiceDefinitions in the manifest apply path now passes
  svc.tls.peer.label() through.
- serviceDefinitionFromRecord parses the textual peer_mode back into
  the enum so registry → snapshot → Upstream carries it cleanly.

with this in place: a manifest with tls.peer = 'require' on a service
flows end-to-end — orchestrator writes 'require' to the services row,
the registry snapshot reads it back as .require, resolveUpstream
copies it onto Upstream, and forwardSingleAttempt dispatches mtls
upstreams to client_dial (the changes from the previous commits).

one new round-trip test (syncConfig persists peer_mode) confirms the
column behaves and updates apply.
@kacy
kacy merged commit 7683f9c into main Jun 23, 2026
10 checks passed
@kacy
kacy deleted the feat/mtls-proxy-outbound branch June 23, 2026 00:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant