Explicit Connection Lifecycle States for TCPConnection

[SeanTAllen]

Explicit Connection Lifecycle States for TCPConnection

Problem

TCPConnection encodes its connection lifecycle as boolean flags: _connected, _closed, _shutdown, _shutdown_peer, plus SSL-related flags _ssl_ready, _ssl_failed, _tls_upgrade, _ssl_auth_failed. These flags form an implicit state machine where valid combinations aren't documented and invalid combinations aren't prevented.

This creates several maintenance challenges:

1. Dense dispatch in _event_notify: The method is ~95 lines with nested branching on event is _event, not _connected and not _closed, and platform ifdef blocks. Different lifecycle phases (client doing Happy Eyeballs, open connection doing I/O, connection shutting down) are interleaved in the same method.

2. Complex callback routing in hard_close(): Which lifecycle callback fires depends on a multi-way branch: SSL present? SSL ready? TLS upgrade? Client or server? Auth failed? These checks inspect 5+ flags to determine the correct callback. The method is ~118 lines.

3. Scattered state transitions: _connected = true is set in _event_notify (Happy Eyeballs success) and _complete_server_initialization. _closed = true is set in hard_close() (both the connecting and connected branches), _close(), and _complete_server_initialization. Understanding the lifecycle requires tracing all flag mutations across a 1644-line file.

Inspiration: ponylang/postgres

The ponylang/postgres library addresses the same class of problem with state objects: each lifecycle state is a separate class, and the Session actor holds a single var state: _SessionState field. Every incoming event is delegated to whatever state object is current. The session actor itself contains no conditional logic about what state it's in — it is purely a dispatcher.

A trait hierarchy provides defaults for operations that are illegal in each state. Capability traits implement valid behavior; denial traits panic on impossible transitions. Each concrete state mixes in the appropriate combination, so it only implements the methods that are valid transitions. Everything else panics by default.

This pattern also ties data ownership to state — per-query accumulation data lives inside the in-flight query state object and is structurally destroyed on state transition.

Analysis: Which of Lori's state machines benefit?

TCPConnection has six interleaved concerns:

Concern	Flags/Fields	Candidate?
Connection lifecycle	_connected, _closed, _shutdown, _shutdown_peer	Yes — gates most operations, has complex dispatch
SSL/TLS	_ssl, _ssl_ready, _ssl_failed, _tls_upgrade, _ssl_auth_failed	Later — tightly coupled to lifecycle, but benefits from lifecycle being clearer first
Backpressure	_throttled, _writeable	No — flow control within the open state
Read flow	_readable, _muted, _yield_read	No — read loop conditions, not lifecycle gates
Happy Eyeballs	_inflight_connections, _had_inflight	Absorbed into the connecting state
Idle timeout	_timer_event, _idle_timeout_nsec	No — simple lifecycle (arm/reset/cancel) tied to the open state

Read flow was evaluated as a candidate. _readable is checked in exactly one place — the _read() loop condition. _yield_read is a one-shot flag cleared by the yield check. _muted suppresses reads but doesn't make operations structurally illegal. All three remain as flags.

Idle timeout adds a timer event handled at the top of _event_notify, but the lifecycle is straightforward: arm when connected, reset on I/O activity, cancel on close. No complex state transitions.

Conclusion: The connection lifecycle is the right starting point. It's where operations become structurally illegal (you can't send on a closed connection), where dispatch is most complex (_event_notify, hard_close()), and where flag combinations are hardest to reason about.

Proposed Design

States

_ConnectionNone ──(_finish_initialization)──► _ClientConnecting  (client path)
                ──(_finish_initialization)──► _Open              (server path)
                ──(_finish_initialization)──► _Closed            (SSL creation failed)

_ClientConnecting ──(connection established)──► _Open
                  ──(hard_close())───────────► _Closed
                  ──(all attempts failed)────► _Closed
                  ──(close(), then all drain)─► _Closed

_Open ──(graceful close)──► _Closing
      ──(hard close)──────► _Closed

_Closing ──(shutdown complete)──► _Closed
         ──(hard close)─────────► _Closed

_ConnectionNone: The universal initial state. All constructors (including none()) start here. No behaviors can fire between construction and _finish_initialization (it's the first behavior queued), so _ConnectionNone is safe as the starting state. When _finish_initialization fires, it transitions to the appropriate state. For TCPConnection.none() (the field initializer placeholder), _finish_initialization is never called because the actor's constructor body replaces the placeholder before any behaviors fire. Every operation on _ConnectionNone is _Unreachable().

_ClientConnecting: Happy Eyeballs in progress. Handles incoming connection-attempt events via foreign_event(). Contains the _connecting_callback() logic — firing _on_connecting(count) as attempts resolve. send() returns SendErrorNotConnected. start_tls() returns StartTLSNotConnected.

When hard_close() is called during connecting, it marks the connection closed, disposes SSL if present, fires _on_connection_failure() with the appropriate reason (ConnectionFailedDNS or ConnectionFailedTCP based on _had_inflight), cancels the idle timer, and transitions to _Closed. Straggler events arriving after hard_close() are cleaned up by _Closed.

When close() is called during connecting (and not muted), a _pending_close flag is set. Subsequent Happy Eyeballs events go to straggler cleanup instead of attempting to establish a connection. When all inflight connections have drained, _on_connection_failure() fires and the state transitions to _Closed. (The _pending_close flag is a localized boolean within _ClientConnecting — it introduces a mini implicit state within this one state class. The trade-off is acceptable: splitting _ClientConnecting into _ClientConnecting and _ClientConnectingClosed would double the connecting logic for a simple binary choice.)

When all connection attempts fail naturally (without explicit close/hard_close), _connecting_callback() calls hard_close(), which handles the cleanup and transition to _Closed.

_Open: Connected and alive. Handles own-event I/O via own_event(), send, close, start_tls. Straggler Happy Eyeballs events (connection attempts that resolved after we picked a winner) are cleaned up in foreign_event(). SSL handshake progress and flow control flags operate within this state.

_Closing: Graceful close initiated. No new sends accepted (is_open() returns false). Sends FIN via PonyTCP.shutdown() once inflight stragglers have drained (_inflight_connections == 0). Continues reading to detect peer FIN and handling write completions. Transitions to _Closed via hard_close() when both local and peer shutdown are complete.

_Closed: Terminal. send() returns SendErrorNotConnected. close() and hard_close() are no-ops. Straggler events are cleaned up (unsubscribe, close fd, destroy).

is_open() / is_closed() semantics: is_open() means "the connection can accept application operations (send, start_tls)." is_closed() means "the connection is no longer accepting application operations." These are not exact complements — during _ClientConnecting, both return false. _Closing returns is_open() = false, is_closed() = true, matching current behavior where _closed = true after _close(). This matters because _register_spawner checks is_closed() to decide whether to store the spawner or immediately notify the listener.

State	is_open()	is_closed()
_ConnectionNone	false	false
_ClientConnecting	false	false
_Open	true	false
_Closing	false	true
_Closed	false	true

Client/server dimension

Currently encoded as the type of _lifecycle_event_receiver (ClientLifecycleEventReceiver vs ServerLifecycleEventReceiver). The _ClientConnecting state is inherently client-only. Within _Open and _Closing, the client/server distinction only affects callback names (_on_connected vs _on_started, _on_connection_failure vs _on_start_failure). State objects match on the receiver type for these differences. No _ClientOpen/_ServerOpen split — it would duplicate nearly identical classes for a small callback routing difference.

Server initialization

Servers receive an fd from the listener and transition directly to _Open during _complete_server_initialization. If SSL session creation failed in the constructor (_ssl_failed), the server transitions to _Closed instead, closing the fd and firing _on_start_failure(StartFailedSSL).

Initialization dispatch

_finish_initialization() is a behavior on TCPConnectionActor that fires asynchronously after construction. All constructors (including none()) start with _state = _ConnectionNone. No behaviors can fire between construction and _finish_initialization (it's the first behavior queued), so _ConnectionNone is safe as the universal initial state.

When _finish_initialization fires, TCPConnection's _finish_initialization() method performs the transition:

• Client path: Checks _ssl_failed (transitions to _Closed with _on_connection_failure(ConnectionFailedSSL) if true). Otherwise transitions to _ClientConnecting and calls PonyTCP.connect() to initiate Happy Eyeballs. Note: the current code does not set _closed = true on client SSL failure (unlike the server path, which does). The state-object design fixes this — both paths transition to _Closed.
• Server path: Checks _ssl_failed (transitions to _Closed with _on_start_failure(StartFailedSSL) if true). Otherwise transitions to _Open, creates the ASIO event, arms the idle timer (if configured), and starts reading.

The _finish_initialization method itself remains on TCPConnection — it's not dispatched through the state interface. It's the bootstrap that sets the first real state.

For TCPConnection.none() (the field initializer placeholder), _finish_initialization is never called because the actor's constructor body replaces the placeholder with a real TCPConnection before any behaviors fire.

State interface

trait _ConnectionState
  fun ref own_event(conn: TCPConnection ref, flags: U32, arg: U32)
  fun ref foreign_event(conn: TCPConnection ref, event: AsioEventID,
    flags: U32, arg: U32)
  fun ref send(conn: TCPConnection ref,
    data: (ByteSeq | ByteSeqIter)): (SendToken | SendError)
  fun ref close(conn: TCPConnection ref)
  fun ref hard_close(conn: TCPConnection ref)
  fun ref start_tls(conn: TCPConnection ref, ssl_ctx: SSLContext val,
    host: String): (None | StartTLSError)
  fun ref read_again(conn: TCPConnection ref)
  fun is_open(): Bool
  fun is_closed(): Bool

_event_notify dispatch

TCPConnection's _event_notify handles timer events and event ownership dispatch before delegating lifecycle-relevant events to the state:

// TCPConnection
fun ref _event_notify(event: AsioEventID, flags: U32, arg: U32) =>
  // Timer events are orthogonal to lifecycle state. Stale timer
  // disposables never reach here because _cancel_idle_timer() clears
  // _timer_event synchronously.
  if event is _timer_event then
    _fire_idle_timeout()
    return
  end

  if event is _event then
    _state.own_event(this, flags, arg)
  else
    _state.foreign_event(this, event, flags, arg)
  end

  if AsioEvent.disposable(flags) then
    PonyAsio.destroy(event)
    if event is _event then
      _event = AsioEvent.none()
    end
  end

Disposable handling is uniform across all states, so it stays on TCPConnection as a post-dispatch step rather than being duplicated in every state. Placing it after state dispatch (not before with a return) ensures that readable/writeable flags in the same notification are still processed by the state first.

This replaces the current nested if event is _event / if not _connected and not _closed structure with per-state dispatch:

• _ClientConnecting.foreign_event(): Happy Eyeballs resolution — the current if not _connected and not _closed block. When _pending_close is set, routes to straggler cleanup instead of connection establishment.
• _Open.own_event(): Normal I/O — the current if event is _event block for an established connection.
• _Open.foreign_event(): Straggler Happy Eyeballs cleanup (unsubscribe, close fd).
• _Closing.own_event(): I/O with shutdown tracking. Same as _Open but reads may complete the shutdown sequence.
• _Closing.foreign_event(): Straggler cleanup. Stragglers are asynchronous and can arrive after close().
• _Closed.own_event() / foreign_event(): Straggler cleanup only (unsubscribe, close fd, destroy disposable events). After hard_close(), the main event has been unsubscribed and the fd closed, but stragglers and disposable notifications may still arrive.
• _ConnectionNone: _Unreachable() for both — no events can arrive before _finish_initialization.

_try_shutdown() is absorbed into _Closing — it only does meaningful work in that state (in all others it early-returns because is_closed() is false). _Closing.own_event() and _Closing.foreign_event() call the shutdown logic after handling I/O or straggler cleanup. The connecting-phase close branch that currently lives in _try_shutdown() (if _connected then ... else ...) is handled by _ClientConnecting._pending_close instead, so it doesn't migrate to _Closing.

Asynchronous operations and state safety

Several behaviors on TCPConnectionActor dispatch to TCPConnection methods that are state-dependent:

• _read_again: Calls _read() (POSIX) or _windows_resume_read() (Windows) to re-enter the read loop. Can arrive after the connection has closed (it's asynchronous). Currently, _read() on a closed connection exits immediately — hard_close() sets _readable = false and _shutdown_peer = true, so the outer loop condition while _readable and not _shutdown_peer is false. _windows_resume_read() guards with if not _connected then return end. Both guards work but rely on flags that the state machine replaces. With state objects, _read_again dispatches through the state interface via read_again() — _Closed no-ops, _Open/_Closing call the platform-specific read method.

• _notify_sent / _notify_send_failed: These fire deferred callbacks via _fire_on_sent() / _fire_on_send_failed() on TCPConnection. They can arrive after close and must still deliver their callbacks (by design — _on_send_failed is expected after _on_closed). They remain on TCPConnection, not gated by state.

• _register_spawner: Server-only. Currently gates on _closed. With state objects, it can use _state.is_closed() directly on TCPConnection without being part of the state interface.

Other state-dependent methods

keepalive() currently gates on _connected (a minor inconsistency with the other methods that use is_open()). With state objects, it gates on _state.is_open(), making it consistent. It remains on TCPConnection — not part of the state interface.

Socket options (set_nodelay(), get_so_rcvbuf(), set_so_rcvbuf(), get_so_sndbuf(), set_so_sndbuf()) currently gate on is_open(). Same treatment as keepalive() — they stay on TCPConnection, gating via _state.is_open().

idle_timeout() currently gates on _connected and not _closed. With state objects, it gates on _state.is_open(). The timer arm/reset/cancel methods remain on TCPConnection, with arm points in initialization and cancel in hard_close().

Read buffer methods (set_read_buffer_minimum(), resize_read_buffer(), expect()) are not state-gated — they can be called anytime. They stay on TCPConnection unchanged.

close() and the mute check

close() currently checks _muted and delegates to hard_close() (if muted) or _close() (if not). This check stays on TCPConnection as a pre-dispatch step — it's the same in every state and doesn't need per-state logic:

fun ref close() =>
  if _muted then
    hard_close()
  else
    _state.close(this)
  end

Each state's close() only handles the graceful (non-muted) path. _Open transitions to _Closing. _ClientConnecting sets _pending_close. _Closing and _Closed no-op. hard_close() is dispatched through the state interface separately.

Spawner notification

Server connections notify their spawning listener via spawner._connection_closed() during teardown (currently at the end of hard_close()). This stays in the hard_close() implementations on _Open and _Closing, since it only applies when a connected server connection closes. _ClientConnecting.hard_close() and _Closed.hard_close() don't need it.

State transition ordering

hard_close() currently sets _connected = false and _closed = true at the top (before firing any callbacks). The state-object equivalent must preserve this: set _state = _Closed before firing _on_closed(), _on_connection_failure(), etc. This ensures that if a callback re-enters TCPConnection (e.g., calling send()), it sees the closed state and gets the appropriate error.

Data ownership

Recommended: states are logic-only. All data (_fd, _event, _read_buffer, _pending_data, SSL fields, flow control flags, idle timeout fields, client startup fields) stays on TCPConnection. State objects provide dispatch and operation gating. Transitions are _state = _Open rather than flag mutations.

The alternative — states owning per-state data — was considered:

• _ClientConnecting could own _host, _port, _from, _ip_version, _inflight_connections, _had_inflight (only relevant while connecting). But these are set in the constructor and consumed in _finish_initialization — they'd need to survive the constructor-to-behavior gap on TCPConnection regardless and then be passed to _ClientConnecting's constructor.
• _Closing could own _shutdown, _shutdown_peer (only relevant during graceful close). But _Open and _Closing share almost all other data (fd, event, buffers, SSL, idle timeout), making ownership between them awkward.

Data ownership can be added incrementally later where it proves valuable. The first iteration focuses on making lifecycle transitions explicit.

Platform differences (IOCP)

POSIX and IOCP have different I/O models (edge-triggered notification vs completion callbacks) but the same state transitions. The platform difference is in how I/O is performed within a state, not which states exist.

With logic-only state objects, platform-specific I/O methods stay on TCPConnection where they are today (_read() for POSIX, _read_completed() and _windows_resume_read() for Windows). State objects call into these methods, and the ifdef blocks remain in the I/O layer.

SSL

SSL remains as fields on TCPConnection for this iteration. The _ssl, _ssl_ready, _ssl_failed, _tls_upgrade, _ssl_auth_failed, _ssl_expect fields and associated logic in _ssl_poll() stay unchanged.

A future iteration could model SSL as a sub-state machine within _Open (e.g., NoSSL | SSLHandshaking | SSLReady). This would simplify the callback routing in hard_close() — the SSL sub-state would know whether to fire _on_connection_failure, _on_tls_failure, or _on_closed. But that's a separate step that builds on the foundation this change lays.

Trait hierarchy

The postgres library uses a rich hierarchy of capability traits and denial traits. This pays off when there are ~20 methods and many states need the same defaults.

For Lori's smaller surface (~8 methods, 5 states), each state can directly implement the full interface. Illegal operations are one-liners:

class _Closed is _ConnectionState
  fun ref send(conn: TCPConnection ref,
    data: (ByteSeq | ByteSeqIter)): (SendToken | SendError)
  =>
    SendErrorNotConnected

  fun ref close(conn: TCPConnection ref) => None
  fun ref hard_close(conn: TCPConnection ref) => None

  fun ref start_tls(conn: TCPConnection ref, ssl_ctx: SSLContext val,
    host: String): (None | StartTLSError)
  =>
    StartTLSNotConnected

  fun is_open(): Bool => false
  fun is_closed(): Bool => true
  // ...

If the method count grows (e.g., when SSL becomes a sub-state machine), a trait hierarchy can be introduced then.

What changes

1. _event_notify splits from one ~95-line method into per-state handlers, each handling only its relevant events. Timer handling stays in TCPConnection as a pre-dispatch step.
2. send(), close(), hard_close(), start_tls() gate on state type rather than boolean flag checks.
3. State transitions become explicit assignments (_state = _Open) rather than scattered flag mutations.
4. hard_close() callback routing is partially simplified — the state knows whether we're in the connecting phase, open, or closing.
5. _read_again dispatches through the state interface via read_again(), preventing reads on closed connections.
6. close() during connecting is handled by _ClientConnecting with a _pending_close flag rather than by setting _closed = true and relying on _try_shutdown() to notice.

What stays the same

• All flow control flags (_readable, _writeable, _throttled, _muted, _yield_read) remain on TCPConnection.
• SSL fields and logic remain on TCPConnection.
• Idle timeout fields and methods remain on TCPConnection.
• I/O methods (_read, _read_completed, _windows_resume_read, _send_pending_writes, etc.) remain on TCPConnection.
• Socket option methods (set_nodelay, buffer sizes) remain on TCPConnection.
• Read buffer sizing methods and expect remain on TCPConnection.
• _try_shutdown() is absorbed into _Closing (see _event_notify dispatch section).
• The public API of TCPConnection is unchanged.
• TCPConnectionActor trait signature is unchanged. The _read_again behavior body changes to dispatch through the state interface instead of calling _read() / _windows_resume_read() directly, but the behavior's public signature stays the same.
• Lifecycle event receiver traits are unchanged.

Pony implementation notes

State objects must be class (not primitive) because the _ConnectionState trait uses fun ref methods. While states like _Closed and _ConnectionNone have no per-instance data and could theoretically be primitives, primitives are val — they can't satisfy fun ref method signatures. Since the trait needs uniformity across all states (and _ClientConnecting needs ref for _pending_close), all states use class. The allocation cost is minimal — state transitions are infrequent relative to I/O operations.

[SeanTAllen]

Implemented in #220.