service: CTP controller-assigned process identity (RFC, alt to #36876)

[jasonhernandez]

Draft / RFC. Alternative to #36876 for the CTP host-check question, per review discussion.

Problem

The distroless migration (#36099) deletes clusterd/ci/entrypoint.sh, which set CLUSTERD_GRPC_HOST=$(hostname --fqdn). That value fed the CTP handshake's server_fqdn check (src/service/src/transport.rs), which rejects a controller connection whose dialed host does not match the replica's own FQDN. So #36099 silently disables that guard on Kubernetes, and #36876 proposed removing it outright.

Review (thanks @def-) pushed back: the check is a real correctness guard, not just defense in depth. The controller builds an ordered replica address list and the Partitioned client uses the connection index to route command shards (compute sends all commands to shard 0), so a connection that lands on the wrong process can corrupt the partition mapping, and the new connection cancels the legitimate one. Kubernetes runs replicas with zero termination grace and reused pod identities, so stale DNS / pod-IP reuse are realistic.

Why not just re-plumb the FQDN

We could feed the pod FQDN back to distroless clusterd (downward API / an arg) and keep the existing check. But the mechanism is vestigial: the command transport is CTP (a custom bincode-over-TCP protocol), not gRPC. The --grpc-host name is a leftover from when it was gRPC. Since we own CTP, we can carry a proper controller-assigned identity instead of a DNS hostname.

This change

Replace the server_fqdn string in the CTP Hello with a controller-assigned identity token, verified per address slot. The handshake compare logic is unchanged (symmetric: reject when both sides present a value and they differ). Only the value and how each side obtains it change:

• The controller assigns each replica process the identity <cluster_id>/<replica_id>/<ordinal> and expects that exact value for the corresponding address slot (src/controller/src/clusters.rs).
• clusterd advertises the same identity in its handshake (src/clusterd/src/lib.rs). The controller passes the replica-scoped prefix <cluster_id>/<replica_id> via --ctp-identity; clusterd appends its own --process ordinal. (The ordinal cannot be baked into a per-pod arg because Kubernetes StatefulSet pods share one arg template.)
• ClusterReplicaLocation carries a ctl_identities list parallel to ctl_addrs, so the expected identity flows from launch to connect.
• --grpc-host / CLUSTERD_GRPC_HOST is replaced by --ctp-identity / CTP_IDENTITY.
• Everything stays Option: None on either side disables the check, preserving current behavior for the process orchestrator and unmanaged replicas.

This restores the guard for managed replicas and retires the misleading grpc naming.

Open design questions (the point of the draft)

1. Generation / incarnation. The token is <cluster>/<replica>/<ordinal> with no incarnation component. It catches wrong-replica and wrong-ordinal misrouting, but not a same-identity stale pod (an old incarnation of the same replica/ordinal reachable at a reused IP). @def-'s suggestion was to include a generation. Closing this needs a durable incarnation source that both the controller's expected value and the pod's advertised value derive from. Where should that live (replica config generation? a controller-assigned per-launch epoch)?
2. Token composition split. clusterd assembles the ordinal onto a controller-supplied prefix (forced by the shared StatefulSet arg template). Acceptable, or do we want a fully controller-formed per-pod token (larger orchestrator change)?
3. Coordination. This supersedes clusterd: remove unused CTP server-FQDN validation #36876 and resolves @def-'s finding on containers: switch environmentd and clusterd to distroless base image #36099. If we take this route, clusterd: remove unused CTP server-FQDN validation #36876 is dropped and containers: switch environmentd and clusterd to distroless base image #36099 no longer needs to re-plumb the FQDN.

Test plan

• cargo check --workspace
• cargo test -p mz-service --test transport (incl. renamed test_handshake_identity_mismatch)
• cloudtest / managed-replica connect on kind
• decide the generation question above before this leaves draft

🤖 Generated with Claude Code