Replace CCF's node-to-node protocol with TLS

[Eddy Ashton (eddyashton)]

Proposal: replace CCF's node-to-node protocol with TLS

CCF currently uses a custom cryptographic protocol for communication between nodes.
We refer to this currently supported protocol as legacy throughout this proposal.
It authenticates consensus headers and encrypts forwarded client requests and responses.
We propose replacing it with mutually authenticated TLS using the existing service-endorsed node identities.
This simplifies the transport and lets us rely on a standard implementation for connection security.
TLS also provides a clean path to post-quantum cryptography as support becomes available in the TLS implementation.
A bridge release supports both transports, preserving CCF's existing live-interoperability guarantees and supported upgrade path.

Configuration

Keep the existing interface for legacy and add a sibling interface for TLS.
The presence of each interface enables that transport.
The following examples show the relevant part of a node's configuration.

Current releases: legacy

{
  "network": {
    "node_to_node_interface": {
      "bind_address": "10.0.0.4:8081"
    }
  }
}

Bridge release: both transports

{
  "network": {
    "node_to_node_interface": {
      "bind_address": "10.0.0.4:8081"
    },
    "node_to_node_tls_interface": {
      "bind_address": "10.0.0.4:8082"
    }
  }
}

TLS-only deployment

{
  "network": {
    "node_to_node_tls_interface": {
      "bind_address": "10.0.0.4:8082"
    }
  }
}

Both interface objects accept an optional published_address, as the existing interface does.
It tells peers where to connect when that differs from the local bind address.
The TLS peer check verifies the expected node identity.

Transport selection

Nodes use the published legacy and TLS addresses recorded for each peer in the replicated node table to choose how to connect.
Each node pair selects its transport independently, according to the endpoints enabled on both nodes:

Endpoints shared by a pair of nodes	Transport used
Both have a TLS endpoint	TLS
Otherwise, both have a legacy endpoint	Legacy
Neither transport is shared	Incompatible pair

Each node considers its own configuration as well as its peer's endpoints.
A missing node_to_node_tls_interface disables TLS for both incoming and outgoing connections on that node, even if its version implements TLS.
A connection failure does not trigger a switch to the other transport.
TLS-first selection exercises the long-term transport in dual-enabled networks.
Legacy, as the protocol not chosen by default for those pairs, needs explicit test coverage.

Node capabilities and service interoperability

Version determines which transports a node implements; configuration determines which of its supported listeners are enabled.

Node version	Legacy	TLS
Earlier 7.x	Yes	No
7.final	Yes	Yes
8.x	No	Yes

A 7.final node participating in an upgrade enables both endpoints.
Applying the selection rule gives the following behavior for supported mixed-version services:

Nodes present in a supported service	Transport between peers
Earlier 7.x and 7.final	Legacy for pairs involving earlier 7.x; TLS between dual-enabled 7.final nodes
7.final and 8.0	TLS for every pair
8.x and 8.y	TLS throughout

These combinations support normal operation and changes of primary.
TLS endpoint information must remain available through membership updates during these mixed-version phases.

Traffic carried over TLS

The legacy channel carries consensus traffic and forwarded client requests and responses.
TLS carries the same traffic, while ledger encryption and integrity protection remain unchanged.
Forwarding remains supported because downstream SDKs have not yet completed its deprecation; its eventual removal can be handled separately.

We propose using separate TLS connections for consensus and forwarding, sharing one TLS listening address:

flowchart LR
    A["Node A"] -->|"Consensus TLS connection"| B["Node B: one TLS listening address"]
    A -->|"Forwarding TLS connection"| B

Loading

This prevents a large forwarded request or response from sitting ahead of consensus messages in the same TCP stream.
The current shared channel's ordering between these traffic types is incidental.
The two TLS connections operate in parallel, with no ordering between them.

To share one port, the receiving node needs to identify each connection's purpose.
Use ALPN (Application-Layer Protocol Negotiation): during the TLS handshake, the nodes select a label meaning "CCF consensus" or "CCF forwarding".
Each connection then carries that traffic type.

Design questions

Why sibling legacy and TLS interfaces?

They preserve the existing configuration and give each transport its own bind and published addresses.
Enabling TLS is an explicit configuration choice, and retiring legacy later means removing its interface.

How does the bridge release obtain its TLS endpoint?

The operator configures it explicitly, including the published address and network access where needed.
Before introducing 8.0 nodes, all 7.final nodes have TLS enabled.
TLS-first selection allows the operator to exercise TLS across the service while it is still entirely on 7.final.

Why one TLS listening port for both traffic types?

Separate connections provide parallel streams while one listener keeps deployment configuration simple.
ALPN identifies each connection's purpose during the handshake.
Application-specific ALPN identifiers are conventional; CCF should choose stable identifiers for its consensus and forwarding wire protocols.

Does certificate renewal require reconnecting?

No.
CCF's node and service certificate renewal preserves the identity keys, and node-to-node authentication deliberately accepts expired certificates.
Existing connections remain established; subsequent handshakes use the renewed certificates.
Ledger rekeying affects separate keys.
Future live identity-key rotation would need its own design.

[Eddy Ashton (eddyashton)]

A few implementation decisions worth highlighting alongside the configuration and migration proposal:

• Authenticate consensus traffic before it reaches Raft. The channels layer handles authentication consistently for both transports: legacy verifies its per-message authentication tag, while TLS authenticates the connection. Both deliver the same normalized consensus messages to Raft.

• Extend the shared OpenSSL connection engine with outbound support. The engine already used for incoming RPC connections can support outgoing connections too, allowing N2N TLS to reuse its socket I/O, worker scheduling, buffering, and shutdown machinery. RPC and N2N retain their respective authentication policies.

• Preserve the existing legacy implementation behind a coexistence adapter. Keep transport selection and new endpoint handling outside the existing channel/socket implementations. This keeps legacy behavior reviewable and makes eventual removal a deletion of the legacy backend and adapter.

• Assemble replication frames before handing them to TLS. AppendEntries includes ledger bytes read after the corresponding ledger mutations. TLS protects the complete assembled frame; ledger encryption and integrity verification remain independent.

[Eddy Ashton (eddyashton)]

There is a migration weakness in the proposal: TLS configuration on a node is not sufficient to make its TLS endpoint visible to peers. Peer addresses come from node metadata in the replicated KV. The primary accepting a join parses that metadata using its own schema and serializes the fields it understands. An older primary therefore drops the new node_to_node_tls_interface field, even when the joining node is TLS-enabled. Both the joiner and the accepting primary must understand the field for it to be recorded.

This can leave a 7.final node configured for both transports but recorded as legacy-only, making it unreachable from TLS-only 8.0 peers. Upgrading the primary later does not restore the missing field, and choosing legacy-first rather than TLS-first does not fix the missing metadata.

The proposed fix is to land parsing and preservation of the new configuration/metadata field early in 7.x, independently of the TLS transport implementation, with multiple releases before 7.final. This gives the schema support time to reach deployed services before normal node replacement cycles introduce TLS-enabled nodes.

We should also add early, clear compatibility checks when introducing 8.0 nodes: identify peers whose recorded metadata lacks the required TLS endpoint and report an actionable error rather than allow a later consensus-connectivity failure. Upgrade documentation should explain how to check the recorded endpoints and remedy missing TLS support. Services which skipped the preparation window may need another node replacement cycle after a schema-aware primary is in place; merely updating configuration or upgrading that primary does not repopulate previously lost metadata.

[Amaury Chamayou (achamayou)]

We need the following sequence:

1. 7.pre which can parse and crucially re-serialise network configs for joiners with node_to_node_tls_interface. For all other intents and purposes, the field is ignored.
2. 7.final which can still respond and connect on node_to_node_interface but also does:
   a. listen on node_to_node_tls_interface
   b. connect to other nodes on node_to_node_tls_interface, where they define it.
3. 8.0.0 which only needs to listen and connect to node_to_node_tls_interface, and never looks at node_to_node_interface

And networks need to completely upgrade to 7.pre, then completely to 7.final, before finally migrating to 8.0.0. The last transition is standard in https://ccf.dev/main/build_apps/release_policy.html#operations-compatibility, but the first is a one-time, additional requirement.

[Eddy Ashton (eddyashton)]

Requirement: both consensus and forwarding N2N connections must enforce TLS 1.3 and an explicit traffic-key update policy using TLS KeyUpdate, replacing manual key rotation in legacy channels. Updates must occur before the key-usage limit of the negotiated cipher, accounting for TLS records rather than application messages. We should not assume OpenSSL initiates these updates automatically. This is independent of certificate renewal.

[Eddy Ashton (eddyashton)]

Removability is an explicit design goal. Keep the existing legacy channel minimally changed and as a distinct parallel path to TLS, so removing legacy later has minimal impact on the retained TLS code. Similarly, dispatch TLS forwarding separately and early, keeping its connection and request-handling code independent of TLS consensus so it can eventually lift out cleanly.

We should achieve this through component boundaries and independent tests, rather than add a public forwarding_enabled flag that creates another configuration-compatibility obligation. TLS supports both traffic types while forwarding remains supported; its eventual removal can follow a separately agreed deprecation and interoperability plan.