Support secure TCP rendezvous for logged-in clients

[TylonHH]

Problem

Logged-in RustDesk clients (observed with 1.4.9) establish a TCP connection to the OSS rendezvous server, then wait for a secure-channel key exchange that stock hbbs does not provide. The client eventually reports:

Failed to secure tcp: deadline has elapsed

This addresses the OSS-server side of #394.

Changes

• advertise an ephemeral Curve25519 box public key signed with the server identity key
• accept the client's KeyExchange response and derive the shared session key
• encrypt subsequent outbound TCP rendezvous messages and decrypt inbound messages
• reject malformed secure-TCP key exchanges
• retain plaintext behavior for logged-out and older clients
• leave WebSocket rendezvous and hbbr behavior unchanged

Compatibility

The handshake is opportunistic. A client that does not answer with KeyExchange continues over the existing plaintext TCP path, so this should not require a coordinated client update.

Validation

• cargo check --bin hbbs with Rust 1.88 on Debian Bookworm
• git diff --check

The build completed successfully. It reported only warnings already present on the current master branch.

Live validation

The patched hbbs image was deployed in the affected OSS hbbs/hbbr plus third-party API setup. The previously failing logged-in RustDesk 1.4.9 connection succeeded immediately without changing the client ID server, relay server, API server, or public key settings.

Prior art

This implementation was adapted to current master from the proof of concept shared by @kamuzon in issue comment #394, specifically kamuzon/rustdesk-server@ec5956f. Credit also goes to @eltorio for the earlier secure-TCP investigation and patch discussed in that issue.

Greptile Summary

This PR adds opportunistic Curve25519/secretbox encryption to the TCP rendezvous path in hbbs, addressing the Failed to secure tcp: deadline has elapsed error that logged-in RustDesk 1.4.9 clients report against the OSS server. When a server identity key (sk) is configured, the server now sends a signed ephemeral box public key to every new TCP connection; clients that respond with a KeyExchange get a fully encrypted session while older/plain clients fall through to the existing plaintext path.

• The Sink enum gains an Option<Encrypt> slot so the server can encrypt outbound frames once the session key is established; inbound decryption is handled by a separate receive_encrypt local that is set on the same loop iteration the KeyExchange is processed.
• Error handling is asymmetric: a KeyExchange with the wrong key count triggers bail! (hard close, logged only at DEBUG), while a message that parses but is not a KeyExchange silently falls through to plaintext — both branches lack a WARN-level log entry, which will make production diagnostics harder.

Confidence Score: 4/5

Safe to merge for the primary goal of unblocking logged-in 1.4.9 clients; all identified concerns are non-blocking observability and compatibility caveats rather than functional failures.

The key derivation, encryption/decryption wiring, and backward-compat fallthrough all appear correct. The three findings are about missing WARN-level logging on failed handshakes, asymmetric error handling between a structurally wrong KeyExchange (hard close) and an entirely non-KeyExchange first message (silent plaintext fallthrough), and the fact that the unconditional server-initiated KeyExchange offer could confuse older clients that don't expect a server-first frame — none of these represent data loss or a broken session under normal operation.

Files Needing Attention: src/rendezvous_server.rs — specifically the handshake error paths around lines 1248-1260 and the unconditional offer at lines 1229-1240.

Important Files Changed

Filename

Overview

src/rendezvous_server.rs

Adds opportunistic Curve25519 key exchange on the TCP rendezvous path: server sends a signed ephemeral box public key, accepts the client's KeyExchange response, derives a shared secretbox session key, and then encrypts/decrypts all subsequent frames. Backward compatibility is maintained for plaintext clients by falling through when the first message is not a KeyExchange. Minor concerns around logging verbosity, asymmetric error handling for malformed exchanges, and the unconditional key-offer to all TCP connections.

Sequence Diagram

sequenceDiagram
    participant C as RustDesk Client (1.4.9+)
    participant S as hbbs Server

    C->>S: TCP connect
    Note over S: sk configured → gen ephemeral box keypair
    S->>C: "KeyExchange { keys: [sign(ephem_pk, server_sk)] }"
    Note over C: Verify signature with server identity key,<br/>generate secretbox key, encapsulate with server ephem_pk
    C->>S: "KeyExchange { keys: [client_pk, box(sym_key, client_sk, server_ephem_pk)] }"
    Note over S: Decrypt sym_key via Encrypt::decode,<br/>arm receive_encrypt + send_encrypt
    C->>S: "RegisterPeer (encrypted, nonce=1)"
    S->>C: "RegisterPeerResponse (encrypted, nonce=1)"
    C->>S: "PunchHoleRequest (encrypted, nonce=2)"
    Note over S: sink.take() → tcp_punch[addr] = Sink::TcpStream(s, Some(encrypt))
    S->>C: PunchHole (encrypted via tcp_punch send_to_sink)

    Note over C,S: Old / plaintext client path
    participant O as Old Client
    O->>S: TCP connect
    S->>O: KeyExchange offer (ignored by old client)
    O->>S: RegisterPeer (plaintext)
    Note over S: handshake_secret taken but msg is not KeyExchange,<br/>receive_encrypt stays None, handle_tcp in plaintext
    S->>O: RegisterPeerResponse (plaintext)

Loading

Prompt To Fix All With AI

### Issue 1
src/rendezvous_server.rs:1248-1249
**Silent bail without client-visible error or warn-level log**

`bail!` terminates the function immediately and is only surfaced at `log::debug!` via the outer `allow_err!`. In production, where debug logging is typically disabled, a client that sends a `KeyExchange` with the wrong number of keys (version skew, client bug) drops silently — no log at WARN or above, and no error frame sent back to the client before the connection is torn down. Adding a `log::warn!` before the `bail!` would make these handshake failures visible during operations without enabling debug logging across the whole server.

### Issue 2
src/rendezvous_server.rs:1244-1260
**Asymmetric error handling between malformed and absent `KeyExchange`**

When the first message parses as a `KeyExchange` with the wrong key count, the connection is terminated (`bail!`). But when the first message is entirely unparseable as protobuf, or parses as a completely different message type, `handshake_secret.take()` silently consumes the ephemeral secret and the message is forwarded to `handle_tcp` as plaintext — the session continues in plaintext mode.

For a client that is actively attempting the handshake but sends a structurally valid `KeyExchange` with exactly 2 keys whose box decryption fails (wrong server key, network corruption), the `?` on `Encrypt::decode` propagates and closes the connection correctly. The missing middle case is a `KeyExchange` whose protobuf parses but whose `msg.union` is `None` (unknown oneof variant in an older protobuf library): it falls through to plaintext, leaving the client in an uncertain state about whether encryption was established.

### Issue 3
src/rendezvous_server.rs:1229-1240
**Key exchange offer sent unconditionally to every TCP connection when `sk` is configured**

When a server identity key is present, the server transmits a `KeyExchange` frame to every new TCP connection before reading a single byte from the client. Existing plaintext clients that have already sent their first message (e.g., `RegisterPeer`) will eventually also read this server-initiated frame from their receive buffer. If any deployed client version treats an unexpected initial server frame as a protocol error rather than ignoring an unknown oneof variant, the connection would fail — this is a backward-compat risk that is only mitigated by a correct `_ => {}` arm in the client's message-dispatch loop.

The PR description acknowledges this is opportunistic, but it would be worth confirming the oldest client version that is expected to talk to this server handles unknown incoming `RendezvousMessage` types gracefully.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

_{Reviews (1): Last reviewed commit: "support secure TCP rendezvous handshake" | Re-trigger Greptile}

Greptile also left 3 inline comments on this PR.

Summary by CodeRabbit

• New Features

  • Added optional secure transport for recent logged-in TCP connections.
  • TCP clients can now complete a key exchange and use encrypted communication for inbound and outbound messages.

• Compatibility

  • Existing clients continue to work with unencrypted TCP communication when secure transport is unavailable.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 42401efc-ee3b-4f33-86f1-266b1deb1ad6

📥 Commits

Reviewing files that changed from the base of the PR and between 6e7de5b and f1bd02a.

📒 Files selected for processing (1)

• src/rendezvous_server.rs

---

📝 Walkthrough

Walkthrough

The TCP rendezvous channel adds optional encrypted transport for recent logged-in clients through key exchange, encrypts outbound messages, decrypts inbound bytes, and preserves plaintext handling when encryption is unavailable.

Changes

Secure TCP transport

Layer / File(s)	Summary
TCP sink encryption src/rendezvous_server.rs	Sink::TcpStream stores optional Encrypt state, and send_to_sink conditionally encrypts serialized TCP messages.
TCP key exchange and receive pipeline src/rendezvous_server.rs	The TCP listener exchanges ephemeral signed keys when configured, derives encryption state, decrypts inbound bytes, and forwards them to existing TCP handlers while retaining plaintext fallback.

Estimated code review effort: 4 (Complex) | ~40 minutes

Sequence Diagram(s)

sequenceDiagram
  participant TCPClient
  participant RendezvousServer
  participant Encrypt
  participant handle_tcp
  RendezvousServer->>TCPClient: Send signed KeyExchange
  TCPClient->>RendezvousServer: Send client KeyExchange
  RendezvousServer->>Encrypt: Derive shared encryption state
  TCPClient->>RendezvousServer: Send encrypted TCP bytes
  RendezvousServer->>Encrypt: Decrypt inbound bytes
  Encrypt->>handle_tcp: Pass decrypted request bytes

Loading

Suggested reviewers: rustdesk

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly describes the main change: adding secure TCP rendezvous support for logged-in clients.
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}