Push SSL Handshake State into the Connection State Machine

[SeanTAllen]

Problem

_Open represents two different application-level states: "TCP connected, SSL handshake in progress" and "fully connected, ready for application data." Five boolean flags (_ssl_ready, _ssl_failed, _ssl_auth_failed, _tls_upgrade, plus _ssl presence) are checked ad-hoc across TCPConnection. Security-critical invariants (no sends before handshake, no concurrent TLS operations) are enforced by scattered flag checks rather than structural state.

Code that cares about the distinction already has to check flags on top of the state: idle_timeout() checks (not _ssl_ready) and (not _tls_upgrade), set_timer() duplicates the same check, _hard_close_connected() has a multi-way branch reading 4 flags to determine callback routing, and is_writeable() checks _ssl_ready. The fix is to split the SSL handshake phase into explicit states.

Design origin: CLAUDE.md "Future Work" section, Issue #240, Discussion #219.

Design: Two New States

Add _SSLHandshaking (initial SSL from ssl_client/ssl_server constructors) and _TLSUpgrading (mid-stream TLS from start_tls()).

These are separate states rather than one parameterized state because they have fundamentally different semantics: _SSLHandshaking has is_open() = false (application hasn't been notified), while _TLSUpgrading has is_open() = true (application already received _on_connected/_on_started). They also have different callback routing on failure, different start_tls() error returns, and different timer behavior. Two states make these differences structural rather than flag-dependent.

Updated State Diagram

_ConnectionNone -> _ClientConnecting -> _SSLHandshaking -> _Open   (SSL client)
                                     -> _Open                      (plaintext client)
_ConnectionNone -> _SSLHandshaking -> _Open                        (SSL server)
                -> _Open                                           (plaintext server)
                -> _Closed                                         (SSL session creation failed)

_ClientConnecting -> _UnconnectedClosing -> _Closed
_ClientConnecting -> _Closed                                       (hard_close / all failed)

_Open -> _TLSUpgrading -> _Open                                    (STARTTLS success)
      -> _TLSUpgrading -> _Closed                                  (STARTTLS failure)
_Open -> _Closing -> _Closed                                       (graceful close)
_Open -> _Closed                                                   (hard close)

_SSLHandshaking -> _Closed                                         (hard_close / handshake failure / timeout)
_TLSUpgrading -> _Closed                                           (hard_close / upgrade failure)

Non-SSL connections follow the same paths as today. No change for plaintext.

State Property Table

State	is_open()	is_closed()	sends_allowed()	Description
_ConnectionNone	false	false	false	Before _finish_initialization. All methods _Unreachable().
_ClientConnecting	false	false	false	Happy Eyeballs in progress.
_UnconnectedClosing	false	true	false	Draining inflight after close() during connecting.
_SSLHandshaking	false	false	false	TCP connected, initial SSL handshake in progress. Application not notified.
_TLSUpgrading	true	false	false	Established connection upgrading to TLS via start_tls(). Application already notified.
_Open	true	false	true	Application-connected, ready for data.
_Closing	false	true	false	Graceful shutdown, waiting for peer FIN.
_Closed	false	true	false	Terminal.

Trait Interface Changes

Add two methods to _ConnectionState:

fun ref ssl_handshake_complete(conn: TCPConnection ref,
  s: EitherLifecycleEventReceiver ref)
fun sends_allowed(): Bool

ssl_handshake_complete(): Called by _ssl_poll() when SSLReady is reached. Each state handles the transition and callback routing:

State	Behavior
_ConnectionNone	_Unreachable()
_ClientConnecting	_Unreachable()
_UnconnectedClosing	_Unreachable()
_SSLHandshaking	Transition to _Open, cancel connect timer, arm idle timer, fire _on_connected() (client) / _on_started() (server)
_TLSUpgrading	Transition to _Open, fire _on_tls_ready()
_Open	No-op (SSL handshake already complete; ssl.state() returns SSLReady persistently)
_Closing	No-op (entered from _Open, handshake was already complete)
_Closed	No-op

sends_allowed(): Returns true only for _Open. Used by is_writeable() to replace the _ssl_ready check. _TLSUpgrading has is_open() = true but sends_allowed() = false, correctly reflecting that sends are blocked during the upgrade handshake.

Method Tables for New States

_SSLHandshaking:

Method	Behavior	Rationale
own_event	I/O dispatch (same as _Open)	Reads/writes drive the handshake
foreign_event	Straggler cleanup (same as _Open)	Happy Eyeballs stragglers can still arrive
send	SendErrorNotConnected	Application hasn't been notified
close	hard_close()	Application doesn't know about the connection; nothing to drain gracefully
hard_close	_Closed, _hard_close_ssl_handshaking()	Fires _on_connection_failure (client) or _on_start_failure (server)
start_tls	StartTLSNotConnected	Not application-connected
read_again	_do_read_again()	Resume read loop for handshake data
ssl_handshake_complete	See above	Transition to _Open
is_open	false	Application not notified
is_closed	false	TCP socket is live
sends_allowed	false	Handshake in progress

_TLSUpgrading:

Method	Behavior	Rationale
own_event	I/O dispatch (same as _Open)	Reads/writes drive the handshake
foreign_event	Straggler cleanup (same as _Open)	Same as _Open
send	SendErrorNotConnected	Handshake in progress, can't send application data
close	hard_close()	Can't send FIN during TLS handshake; routing through _Closing would lose upgrade context for callback routing
hard_close	_Closed, _hard_close_tls_upgrading()	Fires _on_tls_failure(reason) then _on_closed()
start_tls	StartTLSAlreadyTLS	Already upgrading
read_again	_do_read_again()	Resume read loop for handshake data
ssl_handshake_complete	See above	Transition to _Open
is_open	true	Application was already told about the connection
is_closed	false	TCP socket is live
sends_allowed	false	Handshake in progress

Boolean Flags

Eliminated

• _ssl_ready: Replaced by state transitions. _SSLHandshaking / _TLSUpgrading -> _Open at SSLReady. The _Open.ssl_handshake_complete() no-op handles the persistent SSLReady return on subsequent calls. All 6 read sites replaced by state dispatch.
• _tls_upgrade: Replaced by the state machine. _SSLHandshaking = initial SSL, _TLSUpgrading = upgrade. The state's hard_close() and ssl_handshake_complete() implementations encode the correct callback routing. All 5 read sites replaced.

Kept

• _ssl_failed: Constructor-to-initialization bridge. Set in ssl_client/ssl_server constructors, consumed once in _complete_client/server_initialization(). Orthogonal to runtime states — it's consumed before any handshake state is reached.
• _ssl_auth_failed: Set in _ssl_poll() on SSLAuthFail immediately before hard_close(), read during hard_close() in the same synchronous call chain. Intra-method communication (same pattern as _connect_timed_out). Used by _hard_close_ssl_handshaking() and _hard_close_tls_upgrading() to distinguish TLSAuthFailed vs TLSGeneralError.
• _ssl field: The SSL session object. Needed for I/O.

Code Changes

_ssl_poll() SSLReady branch

Simplifies from flag-checking + callback routing to state dispatch:

| SSLReady =>
  _state.ssl_handshake_complete(this, s)

The if not _ssl_ready then guard is no longer needed — _SSLHandshaking and _TLSUpgrading transition to _Open on the first SSLReady. Subsequent calls land in _Open.ssl_handshake_complete() which is a no-op. The _tls_upgrade routing is no longer needed — the state knows which callbacks to fire.

The SSLAuthFail and SSLError branches are unchanged: they set _ssl_auth_failed (for auth fail) and call hard_close(), which dispatches through the state.

_establish_connection()

Transition to _SSLHandshaking when SSL present (was _Open):

match \exhaustive\ _ssl
| let _: SSL ref =>
  _state = _SSLHandshaking
  _ssl_flush_sends()
  // _on_connected() and idle timer deferred until ssl_handshake_complete
| None =>
  _state = _Open
  _arm_idle_timer()
  _cancel_connect_timer()
  match _lifecycle_event_receiver
  | let c: ClientLifecycleEventReceiver ref =>
    c._on_connected()
  end
end

_complete_server_initialization()

Same pattern — _SSLHandshaking when SSL present:

match \exhaustive\ _ssl
| let _: SSL ref =>
  _state = _SSLHandshaking
  _ssl_flush_sends()
  // _on_started() and idle timer deferred until ssl_handshake_complete
| None =>
  _state = _Open
  _arm_idle_timer()
  s._on_started()
end

_do_start_tls()

Transition to _TLSUpgrading (replaces _tls_upgrade = true; _ssl_ready = false):

_state = _TLSUpgrading   // was: _tls_upgrade = true; _ssl_ready = false
_ssl = consume ssl
_ssl_flush_sends()

_hard_close_connected() split

Extract common teardown into _hard_close_cleanup() (cancel tokens, clear pending buffers, cancel timers, unsubscribe event, close fd, dispose SSL). Then three specialized methods:

• _hard_close_cleanup() — common teardown steps in the exact order they appear in the current _hard_close_connected(): set shutdown flags, fire _notify_send_failed for pending token, clear pending buffers, cancel idle/connect/user timers, unsubscribe event, set unreadable/unwriteable, close fd, dispose SSL. Preserving this order matters — SSL dispose must happen after fd close, and timer cancellation must happen before event unsubscribe.
• _hard_close_connected() — cleanup + _on_closed() + spawner notification. Called from _Open.hard_close() and _Closing.hard_close(). The if not _ssl_ready branch is removed — we can never reach this method with an incomplete handshake.
• _hard_close_ssl_handshaking() — cleanup + callback routing by lifecycle receiver type and timeout flag:
  • Client + _connect_timed_out: _on_connection_failure(ConnectionFailedTimeout)
  • Client: _on_connection_failure(ConnectionFailedSSL)
  • Server: _on_start_failure(StartFailedSSL)
  • Then spawner notification. Server-side SSL connections have a spawner, but the spawner message may not have arrived yet (it's asynchronous via _register_spawner). Must replicate the pattern from current _hard_close_connected() lines 767-778: if _spawned_by is present, call spawner._connection_closed() and clear it; if _spawned_by is None, do nothing — the deferred _register_spawner call handles the closed-before-registration case by checking _state.is_closed(). This two-phase pattern is a race condition that's easy to get wrong — copy it exactly.
• _hard_close_tls_upgrading() — cleanup + _on_tls_failure(reason) (where reason is TLSAuthFailed or TLSGeneralError based on _ssl_auth_failed) + _on_closed() + spawner notification.

_enqueue() guard

Change from is_open() to not _state.is_closed():

fun ref _enqueue(data: ByteSeq) =>
  if data.size() == 0 then return end
  if not _state.is_closed() then
    _pending_data.push(data)
    _pending_writev_total = _pending_writev_total + data.size()
  end

During _SSLHandshaking, _ssl_flush_sends() calls _enqueue() to put handshake protocol data on the wire. With is_open() returning false in the new state, the current guard would silently drop handshake data. not is_closed() allows enqueue in all states with a live TCP socket (_SSLHandshaking, _TLSUpgrading, _Open) while blocking it in closed states. Preserves existing _Closing behavior (both guards are false since _Closing.is_closed() = true).

Note: this guard also becomes true for _ConnectionNone and _ClientConnecting (both have is_closed() = false), widening access compared to the old is_open() guard. This is harmless in practice — _enqueue() is only called from _ssl_flush_sends() and _do_send(), neither of which can be reached during those states. But the widening should be noted in a code comment for future maintainers.

Pre-existing issue: _enqueue() dropping data during _Closing may cause missing TLS close_notify alerts during graceful shutdown. This is not introduced by this change — it's the current behavior. File a separate issue to track.

Simplifications

Method	Current check	After	Why it works
idle_timeout() arming	is_open() + (not _ssl_ready) and (not _tls_upgrade)	is_open() only	_SSLHandshaking.is_open() = false blocks arming; _TLSUpgrading.is_open() = true allows it
set_timer()	is_open() + match _ssl guard	is_open() only	Same reasoning
is_writeable()	is_open() and _writeable + _ssl_ready	is_open() and _writeable and sends_allowed()	State dispatch replaces flag check
_do_send()	match _ssl | if not _ssl_ready	Remove check	_do_send() only reachable from _Open.send(); handshake is always complete

I/O dispatch deduplication

_Open.own_event(), _Closing.own_event(), _SSLHandshaking.own_event(), and _TLSUpgrading.own_event() share identical I/O dispatch logic (~15 lines). Two options:

1. Extract to _dispatch_io_event(flags, arg) helper on TCPConnection — each state delegates to it. Eliminates 4-way duplication.
2. Accept duplication — matches the existing convention where _Open and _Closing already duplicate the logic.

This is a style choice. The plan favors extraction since going from 2 copies to 4 tips the scale, but the decision is left to Sean.

Security Analysis

1. Sends before handshake: Structurally prevented. _SSLHandshaking.send() and _TLSUpgrading.send() return SendErrorNotConnected without reaching _do_send(). The _ssl_ready check in _do_send() becomes dead code and is removed.

2. CVE-2021-23222 protection preserved: The _bytes_in_read_buffer > 0 check remains in _do_start_tls(), which runs synchronously in _Open before transitioning to _TLSUpgrading. During _SSLHandshaking, incoming data goes through _ssl_poll() where ssl.read() returns None (no decrypted application data available during handshake) — it's the SSL library that prevents application data delivery, not the state machine. After ssl_handshake_complete() transitions to _Open within the same _ssl_poll() call, the ssl.read() loop correctly delivers any data that became available with the handshake completion.

3. Downgrade prevention: No transition from _TLSUpgrading back to plaintext _Open. On failure, _TLSUpgrading goes to _Closed only. On close(), _TLSUpgrading does a hard close, preventing the connection from returning to plaintext.

4. Authentication failure routing: _ssl_poll() sets _ssl_auth_failed = true then calls hard_close(). _SSLHandshaking.hard_close() routes to _on_connection_failure/_on_start_failure. _TLSUpgrading.hard_close() routes to _on_tls_failure(TLSAuthFailed) then _on_closed(). Authentication failures cannot be silently ignored.

5. start_tls() preconditions watertight: _SSLHandshaking returns StartTLSNotConnected. _TLSUpgrading returns StartTLSAlreadyTLS. All other non-_Open states return StartTLSNotConnected. The precondition checks in _do_start_tls() are unchanged.

6. Connection timeout coverage: The connect timer covers _ClientConnecting + _SSLHandshaking. Armed in _complete_client_initialization(), cancelled in ssl_handshake_complete(). If timeout fires during _SSLHandshaking, _connect_timed_out is set and hard_close() routes to ConnectionFailedTimeout.

Performance Analysis

Hot path (_Open.own_event() -> read/write): Zero overhead.

• No additional virtual dispatch on the data path. _Open.own_event() delegates to _dispatch_io_event() which is an ordinary method call. The state machine virtual call was already there.
• No per-packet allocation. State objects are allocated at state transitions (once per connection lifecycle phase).
• _do_send() gets simpler: removes the match _ssl | if not _ssl_ready check. One fewer branch in the send hot path.
• is_writeable() trades one match on _ssl for sends_allowed() (virtual call, but _Open returns constant true — trivially optimizable).

State transition cost:

• One additional state object allocation per SSL connection: _SSLHandshaking at TCP connect, _Open at handshake completion. For TLS upgrades: _TLSUpgrading at start_tls(), _Open at completion. For non-SSL connections: zero additional allocations.
• State transitions happen once per connection (or twice for TLS upgrade). Negligible versus millions of I/O dispatches per connection lifetime.

Testing Strategy

Existing tests (must all pass unchanged)

This is a behavior-preserving refactoring for all observable behavior. The critical existing tests:

• _TestSSLPingPong, _TestSSLSendv — initial SSL full lifecycle
• _TestSSLIdleTimeout, _TestSSLIdleTimeoutNotArmedDuringHandshake, _TestSSLIdleTimeoutDeferredArm — timer behavior during handshake
• _TestSSLConnectionTimeoutFires, _TestSSLConnectionTimeoutCancelledOnConnect — connect timeout with SSL
• _TestSetTimerNotOpenDuringSSLHandshake, _TestSetTimerNotOpenDuringSSLHandshakeServer — timer rejection during handshake
• _TestStartTLSPingPong, _TestStartTLSPreconditions — TLS upgrade path
• _TestHardCloseWhileConnecting, _TestCloseWhileConnecting — close during connecting

New tests

1. _TestSSLSendDuringHandshake — SSL client against plain TCP server (handshake stalls); verify send() returns SendErrorNotConnected during initial SSL handshake.

2. _TestSSLCloseDuringHandshake — SSL client against plain TCP server; timer calls close() during stalled handshake. Verify _on_connection_failure(ConnectionFailedSSL) fires (close during _SSLHandshaking routes through hard_close).

3. _TestSSLHardCloseDuringHandshake — SSL client against plain TCP server; timer calls hard_close() during stalled handshake. Verify _on_connection_failure(ConnectionFailedSSL) fires.

4. _TestSSLServerHardCloseDuringHandshake — SSL server accepts from plain TCP client; timer calls hard_close(). Verify _on_start_failure(StartFailedSSL) fires.

5. _TestSSLHandshakeCompleteTransitionsToOpen — SSL client/server with valid certs; in _on_connected callback, assert is_open() == true and send() returns SendToken.

6. _TestStartTLSSendDuringUpgrade — Plaintext connection -> start_tls() against plain TCP server (handshake stalls) -> verify send() returns SendErrorNotConnected.

Counterfactual verification: after each new test passes, temporarily break the assertion to confirm it fires.

Build and test commands

make clean && make ssl=3.0.x    # full clean build + all tests
make ci ssl=3.0.x               # tests + examples + stress tests
make test-one t=TestName ssl=3.0.x  # individual test

Implementation Steps

1. Add ssl_handshake_complete() and sends_allowed() to _ConnectionState trait with implementations in all 6 existing states.
2. (Style choice) Extract _dispatch_io_event(flags, arg) helper on TCPConnection from _Open.own_event().
3. Add _SSLHandshaking state class.
4. Add _TLSUpgrading state class.
5. Extract _hard_close_cleanup() from _hard_close_connected(). Add _hard_close_ssl_handshaking() and _hard_close_tls_upgrading().
6. Simplify _hard_close_connected() — remove the if not _ssl_ready branch.
7. Update _establish_connection() — _SSLHandshaking when SSL present.
8. Update _complete_server_initialization() — _SSLHandshaking when SSL present.
9. Update _ssl_poll() SSLReady branch — ssl_handshake_complete() dispatch.
10. Update _do_start_tls() — transition to _TLSUpgrading.
11. Remove _ssl_ready and _tls_upgrade fields and all reads.
12. Simplify _do_send(), is_writeable(), idle_timeout(), set_timer().
13. Update _enqueue() guard to not _state.is_closed().
14. Write new tests.
15. Run full suite: make clean && make ssl=3.0.x.
16. Run CI suite: make ci ssl=3.0.x.
17. Update CLAUDE.md: state machine section (add states, update diagram and table), SSL internals section (describe state-based routing, note flag elimination), TLS upgrade section (describe _TLSUpgrading), idle timeout / user timer sections (simplified gating), Future Work (remove SSL state machine bullet).

Files modified

1. lori/_connection_state.pony — trait additions + 2 new state classes
2. lori/tcp_connection.pony — remove 2 flags, refactor hard_close, update transitions, simplify gating
3. lori/_test_ssl.pony — new tests 1-5
4. lori/_test_start_tls.pony — new test 6
5. lori/_test.pony — register new tests
6. CLAUDE.md — documentation updates

User-facing impact

None. Internal refactoring. The is_open() change during initial SSL handshake (true -> false) is unobservable — no application callback fires during the handshake window, so the application has no entry point to call is_open(). No release notes or changelog entry needed.

Open Questions for Discussion

1. I/O dispatch deduplication: Extract _dispatch_io_event() helper (3 agents recommend), or duplicate across 4 states following existing convention (2 agents recommend)? This is a style choice.

2. _enqueue() during _Closing: Both the old guard (is_open()) and new guard (not is_closed()) block enqueuing during _Closing. This may cause missing TLS close_notify alerts during graceful shutdown. Pre-existing issue, not introduced by this change. File separately?

[SeanTAllen]

Open Question Decisions

1. I/O dispatch deduplication: Extract _dispatch_io_event(flags, arg) helper on TCPConnection. All four states (_Open, _Closing, _SSLHandshaking, _TLSUpgrading) delegate to it.

2. _enqueue() during _Closing: Pre-existing issue, not fixing now. No separate issue filed — noted for future work.

[SeanTAllen]

Remaining Tests

Four tests from the testing strategy are still unimplemented:

1. _TestSSLSendDuringHandshake — SSL client against plain TCP server (handshake stalls); verify send() returns SendErrorNotConnected during initial SSL handshake.
2. _TestSSLCloseDuringHandshake — SSL client against plain TCP server; timer calls close() during stalled handshake. Verify _on_connection_failure(ConnectionFailedSSL) fires.
3. _TestSSLHardCloseDuringHandshake — SSL client against plain TCP server; timer calls hard_close() during stalled handshake. Verify _on_connection_failure(ConnectionFailedSSL) fires.
4. _TestSSLServerHardCloseDuringHandshake — SSL server accepts from plain TCP client; timer calls hard_close(). Verify _on_start_failure(StartFailedSSL) fires.

Tests 5 and 6 from the plan are done (PR #253). Additional tests beyond the original plan have also been added: SSL/TLS handshake failure tests (#253), is_writeable() during handshake tests (#258), and TLSAuthFailed branch test (#261).

[SeanTAllen]

#383 adds the 4 tests