2.3.2

Added

• BREAKING (library API) — feat(health-check): probe port. HealthCheckConfig gains an
  optional port (proto field 9): when set, both HTTP and TCP probes connect to the
  backend's IP address on that port instead of its data port, and the HTTP probe's Host
  (h2c :authority) names that ip:port. Unset, the probe dials the data port as before. It
  serves backends whose health endpoint or companion listener lives on another port, as
  UdpHealthConfig.tcp_port already does for UDP clusters. validate_health_check_config
  refuses a port outside 1..=65535 at the CLI, the TOML loader, the master state and the
  worker. New TOML key port in [clusters.<id>.health_check] and sozu cluster health-check set flag --port; sozu cluster health-check list gains a port column (data when
  unset). port targets a backend IP address: it is refused, with a stable error, when a
  backend of the cluster has a unix_path, in either order and in the TOML loader (DNS-expanded
  entries included), the master state and the worker; a unix_path backend without port is
  probed through its UNIX socket. Library API: HealthCheckConfig and FileHealthCheckConfig gain the public field
  port, so a struct literal must name it (only the protobuf HealthCheckConfig also accepts ..Default::default()). A message or saved
  state without the field decodes as before. port is an additive field 9 on the existing
  AddCluster and SetHealthCheck messages, so it is not fail-closed: a worker older
  than 2.3.2 decodes the message, ignores the unknown field and probes the data port. Upgrade
  the main process and every worker to 2.3.2 before enabling port. Rust source: an
  exhaustive HealthCheckConfig or FileHealthCheckConfig literal must name port
  (Option<u32> in both).

  Before enabling it. Upgrade the main process and every worker, and use the new CLI,
  before setting port: an older worker ignores field 9 and probes the data port, which in
  TCP mode passes every backend accepting connections there; an older sozu CLI that
  patches a cluster by read-modify-write (sozu cluster h2 enable|disable) drops it, and so
  does an older binary loading the saved state. Documented in doc/health_checks.md,
  doc/configure.md and doc/configure_cli.md; covered by
  test_health_check_tcp_mode_probe_port_closed_marks_serving_backend_down,
  test_health_check_http_mode_probe_port_closed_marks_serving_backend_down,
  test_health_check_tcp_mode_probe_port_answering_keeps_backend_up and
  test_health_check_http_mode_probe_port_answering_keeps_backend_up
  (e2e/src/tests/health_check_mode_tests.rs),
  probe_address_replaces_only_the_port_when_one_is_set (lib/src/health_check.rs),
  set_health_check_out_of_range_port_rejected and
  a_health_check_port_and_a_unix_backend_never_share_a_cluster (command/src/state.rs),
  a_health_check_port_and_a_unix_path_backend_never_share_a_file_cluster
  (command/src/config.rs), probe_dials_the_probe_port_not_the_data_port,
  unix_backend_without_probe_port_is_probed_through_its_socket (lib/src/health_check.rs) and
  a_probe_port_already_set_refuses_a_unix_backend and
  a_unix_backend_already_there_refuses_a_probe_port (lib/src/server.rs),
  diff_converges_when_a_probe_port_and_a_unix_backend_swap (command/src/state.rs),
  a_static_reload_converges_when_a_probe_port_and_a_unix_backend_swap and
  a_load_state_converges_when_a_probe_port_and_a_unix_backend_swap
  (bin/src/command/requests.rs), plus unit tests in
  command/src/config.rs and bin/src/cli.rs.

• HTTP and HTTPS frontends can route on request headers
  (#1253). The new
  header_predicates list of a frontend (RequestHttpFrontend field 17, TOML
  header_predicates = [{ name, match, value }], CLI --match-header <match>=<name>[=<value>]) holds conditions that must all be true: exact
  (byte-equal value), regex (anchored at both ends) or present. Names are
  case-insensitive; Host and Cookie are refused. For one hostname, a
  frontend whose predicates hold outranks every frontend without predicates,
  so a canary and its base can share a hostname and path in any declaration
  order. Predicates are part of the frontend's identity, so a removal names
  them too. Frontends without predicates route as before, and the command
  schema change is additive: saved states without the field load unchanged.
  A frontend with predicates travels on dedicated request verbs
  (AddHttpFrontendWithHeaderPredicates & co) that a worker predating the
  feature cannot decode, so it fails the request instead of installing the
  route without its guard; frontends without predicates keep the historical
  verbs. Repeated identical predicates canonicalize to one identity, a TCP
  frontend rejects header_predicates instead of ignoring them, and frontend
  listings show a redacted predicate count rather than names or values.
  Protobuf: RequestHttpFrontend field 17 and Request fields 66-69
  (AddHttpFrontendWithHeaderPredicates, RemoveHttpFrontendWithHeaderPredicates,
  AddHttpsFrontendWithHeaderPredicates, RemoveHttpsFrontendWithHeaderPredicates).
  Rust source: RequestHttpFrontend, HttpFrontendConfig and
  FileClusterFrontendConfig gain the public field header_predicates, so an
  exhaustive struct literal must name it (the protobuf RequestHttpFrontend also accepts ..Default::default());
  request::RequestType gains the four verbs above, so an exhaustive match
  must handle them; ConfigError::InvalidHeaderPredicate { index, reason } is
  new, so an exhaustive match on ConfigError must handle it.
  command::response::HttpFrontend also gains the public field
  header_predicates: Vec<HeaderPredicate>;
  FileClusterFrontendConfig::header_predicates is an
  Option<Vec<HeaderPredicateConfig>>, and HeaderPredicateConfig is a new
  public type (sozu_command_lib::config).
  sozu_lib::router::Router::tree now holds
  (PathRule, MethodRule, HeaderRules, Route) tuples instead of
  (PathRule, MethodRule, Route), and sozu_lib::router::RouterError gains
  InvalidHeaderPredicate { index, reason }, so an exhaustive match on it
  must handle it.

• feat(mux): redirect = "silent_close" refuses a request without any
  response (#767), the
  equivalent of nginx's return 444. A new RedirectPolicy::SILENT_CLOSE = 5
  (appended to the protobuf enum; TOML redirect = "silent_close", CLI
  --redirect silent_close) makes an HTTP/1.x frontend close the connection
  without writing a byte and an HTTP/2 frontend reset only the refused stream
  with RST_STREAM(CANCEL), leaving the connection's other streams running. The
  backend is never contacted, whether or not the frontend names a cluster, and
  the policy wins over the frontend's other redirect, rewrite and authentication
  settings. The access log records status 444 with message silent_close; the
  line is not an error (http.errors is not incremented) and counts in
  http.status.4xx. The main sends a silent_close add as
  AddHttpFrontendWithSilentClose (field 70) or
  AddHttpsFrontendWithSilentClose (field 71); other frontends keep the legacy
  verbs. A worker older than 2.3.2 cannot decode these requests and fails them,
  so it never installs an unguarded forwarding route. Rust source:
  RedirectPolicy gains SilentClose, request::RequestType gains
  AddHttpFrontendWithSilentClose and AddHttpsFrontendWithSilentClose, and
  sozu_lib::RetrieveClusterError gains SilentClose, so an exhaustive match
  on any of them must handle the new variants.

• Backends reachable over UNIX domain sockets
  (#152). A backend's new
  unix_path (configuration file, AddBackend.unix_path in the protocol,
  sozu backend add --unix-path) takes an absolute path or a Linux
  abstract-namespace @name; HTTP, HTTPS and TCP clusters dial it instead of
  address, which stays the backend's identity and must be unique in its
  cluster. Health checks probe the same socket. UDP is not supported: the
  configuration file and the main process's state both refuse it there.
  A UNIX backend travels on a new AddUnixBackend request
  (RequestType::add_backend picks the verb, and the main process rewrites
  every fan-out to it): a main process or worker that predates it cannot decode
  that request, so it never registers the backend or dials address over TCP.
  A client that sends unix_path on the historical AddBackend instead is
  outside this guarantee: an older decoder drops the field. Protobuf: AddBackend field 7
  (unix_path) and Request field 72 (AddUnixBackend). Covered by
  command/tests/unix_backend_verb.rs and the test_*unix* e2e tests
  (e2e/src/tests/unix_backend_tests.rs). Rust source: AddBackend and
  BackendConfig gain the public field unix_path (as does the new FileBackendConfig), so
  an exhaustive struct literal must name it (the protobuf AddBackend also accepts ..Default::default());
  request::RequestType gains AddUnixBackend, so an exhaustive match must
  handle it; ConfigError::InvalidBackendUnixPath { cluster_id, address, reason }
  is new, so an exhaustive match on ConfigError must handle it.
  command::response::Backend and sozu_lib::backends::Backend also gain the
  public field unix_path: Option<String>, which an exhaustive struct literal
  must name.

• A backend address in the configuration file may be a host:port as well
  as a socket address (#347). The name is resolved once, with the system
  resolver, when the file is loaded (at startup or by sozu reload --file);
  it is never re-resolved while running. Each distinct resolved address
  becomes a backend with a deterministic id, <backend_id>-<address> or
  <cluster_id>-<host:port>-<address>; a configured sticky_id is
  suffixed with -<address> so each resolved backend keeps its own sticky
  session, whatever the number of addresses. A name that does not resolve refuses
  the file with an error naming the cluster and the entry. A file without a
  host name loads exactly as before; this sub-change adds no protobuf message,
  field or verb and needs no worker change (the resolved backends reach workers
  as ordinary AddBackend requests). Rust source: FileClusterConfig::backends
  is now Vec<FileBackendConfig> instead of Vec<BackendConfig>; the new
  public FileBackendConfig (address: String, weight, sticky_id,
  backup, backend_id, unix_path) holds the unresolved entry, and
  BackendConfig keeps a resolved SocketAddr. Code that builds or reads a
  FileClusterConfig literal must use FileBackendConfig. The new
  ConfigError::BackendAddressResolution { cluster_id, address, reason }
  variant is returned for a name that does not resolve, so an exhaustive
  match on ConfigError must handle it.

• sozu cluster close-sessions --id <cluster> [--backend-id <id>] [--address <ip:port>]
  closes, on every worker, the HTTP, HTTPS, TCP and WebSocket sessions holding a backend
  connection to a cluster, or to one of its backends (#123). Sessions close the way a soft
  stop closes them: idle HTTP/1.1, TCP and WebSocket sessions at once, an HTTP/1.1 request in
  flight answered with Connection: close first when the backend response headers are not
  parsed yet, an HTTP/2 session after a GOAWAY, bounded by its graceful-shutdown deadline when
  that is nonzero (0 drains without bound, as on a soft stop). The answer counts, per worker, the sessions closed
  and those still draining. The new Request.close_sessions (field 65) and
  ResponseContent.closed_sessions (field 19) are additive; the request is never saved in the
  state nor replayed by load_state or an upgrade. A worker that predates it cannot decode
  it and closes nothing; it does not answer, so the request fails once the command times out.
  The main process and every worker must run this release before it is used.
  Rust source: the generated request::RequestType oneof gains CloseSessions and
  response_content::ContentType gains ClosedSessions, so an exhaustive match on either
  must handle the new variant.

• feat(health-check): protected custom request headers. An HTTP health check, on
  its HTTP/1.1 and h2c probe alike, can send custom request headers such as Authorization
  whose values are credentials kept out of the configuration, the saved state, queries,
  logs and Debug output. TCP mode refuses them. In TOML, repeat
  [[clusters.<id>.health_check.headers]] with exactly name and value_file (no inline
  value; an absolute path is expected); the configuration stores ordered references only,
  and parsing keeps the reference as written and opens no file. On the command line, repeat sozu cluster health-check set --header-file NAME=PATH,
  split at the first =, where PATH=- reads standard input for at most one header. The
  reader removes one final LF and a CR right before it, and nothing else. Headers are
  ordered, at most 32, with unique (case-insensitive) RFC 9110 token names; Host,
  Connection, Keep-Alive, Proxy-Connection, Transfer-Encoding, Content-Length,
  TE, Trailer, Upgrade, Expect and names starting with : are reserved. The sum of
  name + value + 4 bytes is at most 8192 and the h2c HPACK block at most 16384 bytes; an
  invalid value is refused before any effect.

  The protobuf adds HealthCheckConfig.headers (field 10, HealthCheckHeader { name, fingerprint }), HealthCheckHeaderValue, SetProtectedHealthCheck and
  QueryProtectedHealthCheckSupport. The state and the queries hold the ordered names and
  fingerprints sozu-hc-hmac-sha256:v1:<32 lowercase hex epoch>:<64 lowercase hex tag>
  under a memory-only key of the main process; the value_file references live in the
  configuration only and never reach a worker. A change goes through an acknowledged
  Prepare, Commit, Abort and Release exchange with the workers, with the other
  changes to the same cluster fenced meanwhile; this describes the protected health-check
  change only, not a global transaction. An error before every Commit is acknowledged
  retains or restores the previous configuration; after every Commit acknowledgement, a
  refused or timed-out Release is a retention or cleanup failure and cannot roll back a
  before-image the workers already destroyed. A replaced or respawned worker receives and
  acknowledges the values before it is Running or probes. A saved state is authoritative at
  startup, on reload and for LoadState, RemoveCluster included: a foreign or stale epoch
  cannot activate a value by carrying the same cluster name, and a same-name configuration
  entry cannot resurrect a saved removal. A reload that supplies value_file headers
  for an existing protected cluster retains the old health check A through the generic
  AddCluster phase until the acknowledged protected transaction publishes B; an
  authoritative reload without headers invalidates A and aligns the journal with the
  resulting state, and an explicit RemoveHealthCheck or RemoveCluster stays a
  removal. A probe whose header values are absent is not sent and is
  counted by the new health_check.missing_secret counter, which is neither a success nor a
  failure and leaves the backend health unchanged. Every sozu process sets RLIMIT_CORE to
  zero (soft and hard) and, on Linux, PR_SET_DUMPABLE to zero as its first action, a
  per-process protection only. Owned buffers holding a value are zeroized; the allocator, the
  kernel, socket buffers and swap are out of reach. The probe itself is plaintext HTTP/h2c:
  use a dedicated least-privilege probe-only credential and a protected network path to the
  backend. Documented in doc/health_checks.md, doc/configure.md, doc/how_to_use.md,
  bin/config.toml, command/README.md and the sozu-lib crate documentation.

  Before enabling it. Upgrade the main process and every worker to 2.3.2. With
  --header-file, the CLI first requires an exact positive typed capability answer from the
  main process and every worker within a fixed five seconds (also with --timeout 0), and
  opens no file and reads no standard input before it; without --header-file the ordinary
  SetHealthCheck path is unchanged. Field 10 is additive, so a peer older than 2.3.2
  ignores it rather than failing: this preflight is the only guard, and a generic
  sozu-command-lib integrator must implement it before reading or sending a value, as the
  library types do not add it; the hardened main may read the value_file references of
  its configuration into protected memory before it launches the workers, without that
  query. Rust source: an exhaustive HealthCheckConfig or
  FileHealthCheckConfig literal must name headers, and an exhaustive match on
  request::RequestType or response_content::ContentType must handle the new variants.

🔄 Changed

• feat(load-balancing): every policy honours weight
  (#676). ROUND_ROBIN,
  LEAST_LOADED and POWER_OF_TWO ignored a backend's weight; only
  RANDOM, HRW and MAGLEV read it. ROUND_ROBIN now runs smooth weighted
  round robin (nginx's scheme): over every Σw selections a backend of weight
  w is picked exactly w times, interleaved (5, 1, 1 selects
  a a b a c a a). LEAST_LOADED and POWER_OF_TWO compare each candidate's
  load divided by its weight, exactly (cross-multiplied in integers for the
  connection and request counts). With no client key, HRW and MAGLEV fall
  back to the weighted round robin. A 0 or negative weight counts as 1, as
  under HRW and MAGLEV; RANDOM keeps reading the raw value. Behaviour
  change: a cluster that sets unequal weights under one of these three
  policies now spreads its traffic by them. A cluster whose backends all
  carry the same weight (the default, 100, when unset) selects exactly the
  same sequence as before. No command, configuration or state format changes.

• A soft stop, like cluster close-sessions, now marks an HTTP/1.1 request still waiting for
  its response as the last one of its connection: the response carries Connection: close
  when its headers are not sent yet, and the connection closes as soon as it is sent. A
  client no longer has to discover the close after a response that announced keep-alive.

• A backend belongs to its cluster (#996). AddBackend (sozu backend add)
  naming a cluster the state does not hold, never added or already removed,
  is now refused with ClusterNotFound by the main process and by every
  worker, instead of storing a backend no cluster owns. RemoveCluster
  already removes the cluster's backends (#1729), so no backend outlives its
  cluster. Frontends stay independent: one may still be added before its
  cluster. The configuration file, SaveState and ConfigState::diff all
  emit a cluster before its backends and are unaffected. Rust source:
  StateError (sozu_command_lib::state) gains
  ClusterNotFound { kind, cluster_id }, so an exhaustive match on it must
  handle it.

Upgrade notes

• Protected health-check headers (operator summary). Upgrade the main process and every
  worker to 2.3.2, then enable --header-file; the CLI refuses to read a secret otherwise.
  After a cold start the main process holds a fresh memory-only key and epoch. If a restored
  saved state is authoritative for a health check and carries fingerprints of another epoch,
  that health check sends no probe, and counts health_check.missing_secret, until you
  re-inject the exact same ordered headers with --header-file; a name alone never activates
  a value, and a cold SaveState never restores clear values. A hot sozu upgrade between
  2.3.2 mains uses upgrade protocol V3, a memory-only transport of the complete snapshot with
  no tempfile and no memfd, which carries the key, epoch and values without rereading a
  file; 2.3.2 writes V3 only and never falls back to V2, while it still reads an incoming
  2.3.1 (V2) file handoff, and an absent protocol marker is the 2.2.1 legacy form. Rolling back to 2.3.1 is not a hot handoff: reset
  the affected health checks without --header-file, delete their headers tables from the
  configuration, and restart from that configuration and state. A 2.3.1 peer ignores the
  unknown protobuf field 10 and would probe without the headers, and the 2.3.1 TOML parser
  does not understand the new tables, so it refuses a configuration that still holds them.
• Tooling that sends AddBackend before the AddCluster of the same cluster
  now gets an error for each such backend and must add the cluster first.
• A saved state file written by hand, or by tooling, with a backend before
  its cluster or with a backend of a cluster it never adds: LoadState skips
  that backend with a warn! log and counts it in
  config.load_skipped_invalid, so it is not loaded. A state saved by Sōzu
  itself lists every cluster before its backends.
• A worker of this release refuses such a backend even from an older main
  process, which may still hold it in its own state.

Fixed

• 2.3.2 accepts a legacy 2.3.1 saved state that has no certificate_metadata
  when a state diff changes a listener or removes unrelated frontends or
  backends while retaining certificates. The debug-build replay check in
  ConfigState::diff used to compare the replayed state's empty sidecar with
  the target's normalized one and panicked on semantically identical
  certificates; both sides are now normalized the same way. Explicit
  expired_at and insertion-order differences still produce certificate
  operations.
• Re-adding the last HTTP, HTTPS, TCP or UDP listener removed from an address
  now restores the frontends retained in worker configuration state; HTTPS also
  restores the retained certificates. The worker completes this reconstruction
  before acknowledging the listener add and discards only the new listener if
  reconstruction fails. Existing same-address interface siblings keep their
  current routes and certificate resolver without duplicate replay.
• AddCertificate and ReplaceCertificate now reject malformed private keys,
  demonstrably mismatched certificate/key pairs and names the SNI table cannot
  host before mutating main-process state. Workers use the same validator before
  touching the resolver, so a refusal preserves the saved state and the
  certificate already serving traffic. A crypto provider that cannot expose
  the key material returns rustls' InconsistentKeys::Unknown; Sōzu keeps
  accepting that result under rustls' CertifiedKey::from_der contract.
• ReplaceCertificate now resolves an empty names list from the certificate's
  SAN/CN before recording the replacement, matching AddCertificate and making
  saved-state replay stable. Replacements with the same leaf fingerprint update
  the resolver when their canonical certificate material or effective
  expiration differs. A strict retry with identical material and expiration
  preserves the existing wrapper, SNI tie ordering and expiration gauge.
• ConfigState now persists each TLS certificate's effective expired_at
  override and insertion order. SaveState, LoadState, main-process handoff,
  worker bootstrap and worker upgrade therefore preserve both expiration-based
  selection and the insertion-order tie break. The new public
  ConfigState::certificate_metadata sidecar is a Rust source compatibility
  change for downstream exhaustive ConfigState struct literals, which must
  name certificate_metadata; for this sub-change only, the certificate map and
  command protobufs are unchanged. Old JSON without the field remains readable
  and malformed partial metadata is normalized deterministically on the first
  state-changing certificate command. Direct in-memory writes to the still
  public certificates map bypass that invariant and remain unsupported.
• Certificate-state reconciliation now compares material, expiration and
  insertion order. When only order differs, it retains the longest usable
  target prefix and moves only the required suffix with RemoveCertificate
  followed by AddCertificate; a replay after an interrupted pair converges.
  Canonical additions append in O(1); each removal or effective replacement
  compacts one address's ordinals in O(N), so a reconciliation moving N
  certificates can take O(N²) state work.
  This sequence is not atomic: a name carried only by the certificate being
  moved can temporarily fall back or become unavailable. Set replacement can
  also pass through an empty resolver.
• The first hot upgrade from 2.3.1 cannot recover expiration overrides or
  resolver insertion order that the 2.3.1 main process never stored. The new
  main accepts that legacy state and assigns deterministic fallback order with
  no overrides. Until each old worker is upgraded, its retained runtime
  resolver can select a different certificate from the new main. Worker
  replacement converges to the new main and may change the served certificate;
  later saves, replays and upgrades preserve that normalized state. Re-emit
  ReplaceCertificate commands from the authoritative certificate source
  before replacing workers when the lost overrides must remain effective. A
  2.3.1 SaveState file is a separate boundary: it emits certificates in one
  concrete but arbitrary HashMap iteration order, and loading that request
  stream records the order it contains rather than reconstructing history.
• A listener activation refused by the runtime now restores an inactive flag in
  both main and worker configuration state. Activating a listener that was
  already active creates no deactivation inverse. A fan-out timeout remains an
  indeterminate transport result: even when no acknowledgement arrived and the
  main state was restored, a late worker success can still have bound a socket,
  so the operator must restart or reconcile workers before retrying.
• AddCluster now parses its per-cluster HTTP answer templates in the main
  process before replacing configuration state. Initial additions, upserts and
  state replay use the worker runtime's precedence (answers["503"] overrides
  the legacy answer_503 field), so a malformed answer is refused while the
  prior cluster state and live answer remain in place. This gate prevents new
  invalid mutations; it does not repair state already persisted by an older
  version.