Implementation plan: SSL as first-class TCPConnection concern

[SeanTAllen]

Context

This is a concrete implementation plan for Direction A from Discussion #157 (SSL as a first-class TCPConnection concern). It builds on the design decisions captured in that discussion.

Design: #157

Summary of design decisions

• Four explicit constructors: client, server, ssl_client, ssl_server
• SSLContext as dependency injection point; TCPConnection creates SSL session internally
• No changes to ponylang/ssl library
• Send before connected is accepted and queued; every send gets an explicit outcome (_on_sent or _on_send_failed)
• SSL handshake failure maps to _on_connection_failure(); no SSL-specific callbacks
• ALPN results not surfaced; NetSSLLifecycleEventReceiver (on_alpn_negotiated, on_ssl_auth_failed, on_ssl_error) intentionally dropped — these can be added back later in an additive fashion if needed
• DataInterceptor trait and related infrastructure removed
• SendErrorNotConnected and SendErrorNotReady removed (sends are queued instead)

Implementation steps

Step 1: Add _on_send_failed to lifecycle receiver traits

File: lifecycle_event_receiver.pony

Add to both ServerLifecycleEventReceiver and ClientLifecycleEventReceiver:

fun ref _on_send_failed(token: SendToken) => None

This is additive and backwards compatible — existing implementors inherit the default.

Step 2: Add pre-connection send queueing to TCPConnection

File: tcp_connection.pony

Add a queue for sends that arrive before the connection is established:

• New field: a queue of (ByteSeq, SendToken) pairs for pre-connection sends
• Modify send(): when not connected (or SSL handshake in progress), queue the data and return a SendToken instead of returning SendErrorNotConnected or SendErrorNotReady
• On successful connection (and SSL handshake completion): flush the queue through the normal send path
• On connection failure or close with queued sends: fire _on_send_failed(token) for each queued item via the _notify_send_failed behavior on TCPConnectionActor

Decision point: What should send() do after _on_closed or _on_connection_failure has been delivered? Options: (a) return a new error type like SendErrorClosed, (b) accept and immediately schedule _on_send_failed, or (c) silently drop. This needs to be resolved — leaning toward (a) for explicitness since the user already knows the connection is gone.

Also modify the existing pending write buffer behavior: when the connection drops with data in the OS write buffer (_pending), fire _on_send_failed for the associated SendToken if one exists.

Step 3: Add _notify_send_failed behavior to TCPConnectionActor

File: tcp_connection_actor.pony

Add a new behavior parallel to _notify_sent:

be _notify_send_failed(token: SendToken) =>
  _connection()._fire_on_send_failed(token)

And add _fire_on_send_failed to TCPConnection that delivers to the lifecycle receiver.

This ensures _on_send_failed is always delivered in a subsequent behavior turn, never synchronously during send() or close(), matching the pattern established by _on_sent.

Step 4: Remove SendErrorNotConnected and SendErrorNotReady

File: send_token.pony

Remove the SendErrorNotConnected and SendErrorNotReady primitives. SendErrorNotWriteable remains for backpressure. Update the SendError type alias accordingly. If a "send after closed" error is added (Step 2 decision point), add that here.

Step 5: Add SSL constructors to TCPConnection

File: tcp_connection.pony

Add new fields for SSL state:

• _ssl: (SSL ref | None) — the per-connection SSL session
• _ssl_ready: Bool — whether handshake is complete

Add two new constructors:

new ssl_client(auth: TCPConnectAuth, ssl_ctx: SSLContext val,
  host: String, port: String, from: String,
  enclosing: TCPConnectionActor ref,
  ler: ClientLifecycleEventReceiver ref)

Internally calls ssl_ctx.client(host) to create the SSL iso session. Otherwise identical setup to client().

new ssl_server(auth: TCPServerAuth, ssl_ctx: SSLContext val,
  fd': U32,
  enclosing: TCPConnectionActor ref,
  ler: ServerLifecycleEventReceiver ref)

Internally calls ssl_ctx.server() to create the SSL iso session. Otherwise identical setup to server().

Both constructors must handle the case where ssl_ctx.client() or ssl_ctx.server() raises an error (partial function) — the connection should fail gracefully, eventually delivering _on_connection_failure or equivalent.

Step 6: Add internal SSL logic to TCPConnection

File: tcp_connection.pony

Move the logic currently in SSLClientInterceptor and SSLServerInterceptor into TCPConnection private methods. Key additions:

On connection established (in _complete_client_initialization / _complete_server_initialization):

• If _ssl is not None, initiate the handshake: flush initial SSL data (ClientHello for clients) via _ssl.can_send() / _ssl.send() → _send_final()
• Do NOT deliver _on_connected / _on_started yet — wait for handshake

Incoming data (in _deliver_received):

• If _ssl is not None: feed data to _ssl.receive(), then call a new _ssl_poll() method
• _ssl_poll() checks _ssl.state():
  • SSLReady and not yet _ssl_ready: set _ssl_ready = true, deliver _on_connected / _on_started, flush pre-connection send queue
  • SSLAuthFail / SSLError: call hard_close(), which will trigger _on_connection_failure (for clients pre-ready) or _on_closed (for servers or post-ready)
• Read all available decrypted data via _ssl.read() in a loop, deliver each chunk to the lifecycle receiver's _on_received
• Flush any SSL protocol data via _ssl.can_send() / _ssl.send() → _send_final()

Outgoing data (in send()):

• If _ssl is not None and _ssl_ready: encrypt via _ssl.write(), then send encrypted data via _ssl.can_send() / _ssl.send() → _send_final()
• If _ssl is not None and not _ssl_ready: queue in pre-connection queue (Step 2)

On close (in hard_close()):

• If _ssl is not None: call _ssl.dispose()

Expect handling:

• If _ssl is not None: store the application's expect value internally, but tell the TCP read layer to use 0 (read all available). Apply the stored expect value when chunking decrypted data from _ssl.read().

Step 7: Remove DataInterceptor infrastructure

Files: tcp_connection.pony, data_interceptor.pony

• Delete data_interceptor.pony entirely (removes DataInterceptor, WireSender, IncomingDataReceiver, InterceptorControl)
• Remove adapter classes from tcp_connection.pony: _WireSenderAdapter, _IncomingDataReceiverAdapter, _InterceptorControlAdapter
• Remove the interceptor parameter from client() and server() constructors
• Remove all interceptor-related fields: _interceptor, _interceptor_ready, _wire_sender, _incoming_receiver, _interceptor_control
• Remove _interceptor_signal_ready() method
• Remove all interceptor branching from send(), _deliver_received(), is_writeable(), hard_close(), expect(), and initialization methods

Step 8: Remove SSL interceptor classes

File: net_ssl_connection.pony

Delete the file entirely (removes NetSSLLifecycleEventReceiver, SSLClientInterceptor, SSLServerInterceptor). Their logic has been absorbed into TCPConnection in Step 6.

Step 9: Update examples

examples/net-ssl-echo-server/: Replace SSLServerInterceptor usage with TCPConnection.ssl_server() constructor. Remove interceptor creation. The echo server actor's structure stays the same — it still implements TCPConnectionActor & ServerLifecycleEventReceiver.

examples/net-ssl-infinite-ping-pong/: Replace SSLClientInterceptor / SSLServerInterceptor usage with TCPConnection.ssl_client() / TCPConnection.ssl_server() constructors. Remove interceptor creation.

examples/echo-server/ and examples/infinite-ping-pong/: No changes (no SSL).

Step 10: Update tests

_test.pony:

• Update _TestSSLPingPong and related test actors to use ssl_client / ssl_server constructors instead of interceptors
• Add test for send-before-connected: send data immediately after creating the connection, verify it arrives after connection establishes
• Add test for send failure notification: send data, close the connection before it establishes, verify _on_send_failed fires for each queued send
• Update _TestSendAfterClose — it currently checks for SendErrorNotConnected, which is being removed. Update to match the new post-close send behavior (see Step 2 decision point)
• Remove any interceptor-specific test code

Step 11: Update CLAUDE.md

Update the project CLAUDE.md to reflect:

• New constructor signatures
• Removal of DataInterceptor, interceptor interfaces, SSL interceptor classes
• New send queueing behavior and _on_send_failed callback
• Updated architecture description (SSL is internal to TCPConnection, not an interceptor)
• Updated "How to implement a server/client" examples
• Updated "How to add SSL" examples

Ordering and dependencies

Steps 1-4 (send system changes) can be done first as a standalone improvement — they don't require SSL changes. Steps 5-6 (SSL internals) depend on step 2 (pre-connection queue). Steps 7-8 (removal) depend on steps 5-6 and 9-10 (examples/tests updated to new API). Step 11 is last.

A possible phased approach:

• Phase 1: Steps 1-4 (send queueing + failure notification). Can be merged independently. Note: While the interceptor infrastructure still exists during this phase, removing SendErrorNotReady (Step 4) requires changing the interceptor-not-ready code path in send() to queue instead of returning an error. Phase 1 must handle both the !is_open() and !_interceptor_ready cases with queueing.
• Phase 2: Steps 5-6 (SSL constructors + internal SSL logic). Adds new constructors alongside existing interceptor-based approach.
• Phase 3: Steps 7-10 (remove old infrastructure, update examples/tests). Breaking change.
• Phase 4: Step 11 (documentation).

Or it can be done as a single branch if preferred.

[SeanTAllen]

Decision point in Step 2: Add pre-connection send queueing to TCPConnection.

we should return a send error.

[SeanTAllen]

Revised plan: No pre-connection send queueing

After review, the plan has been revised. The pre-connection send queue (original Step 2) is removed, and several design decisions have been resolved. This comment supersedes the send system portions of the original plan body; the SSL integration steps are mostly unchanged.

Design principle: TCPConnection does not queue application data

TCPConnection does not queue data on behalf of the application. The only data TCPConnection holds in its _pending buffer is the unsent remainder of a single accepted send() during backpressure — a partial write that the OS couldn't accept in full. The moment this happens, TCPConnection stops accepting new sends (SendErrorNotWriteable), so only a portion of one accepted send is ever buffered internally.

If the application wants to send data before the connection is established, during backpressure, or in any other situation where send() returns an error, queueing is the application's responsibility. The enclosing actor decides what to do: queue locally, drop, close, etc.

This means:

• Before connection established: send() returns SendErrorNotConnected
• After connection closed: send() returns SendErrorNotConnected
• During backpressure: send() returns SendErrorNotWriteable
• During SSL handshake: send() returns SendErrorNotConnected (connection isn't "connected" from the user's perspective until handshake completes)
• Send accepted then connection drops: _on_send_failed(token) fires

_on_send_failed fires only for data that send() accepted (returned a SendToken) but that couldn't be delivered to the OS — specifically when _pending_token is set (partial write in progress) and hard_close() is called before the pending data drains. Since backpressure prevents concurrent sends, there is at most one _pending_token at any time. No multi-token tracking is needed.

Resolved design decisions

_on_send_failed / _on_closed ordering: _on_closed() fires synchronously during hard_close(). _on_send_failed(token) fires in a subsequent turn (deferred via _notify_send_failed behavior). The user sees _on_closed first, then _on_send_failed. This is consistent with _on_sent — data handed to the OS before a close also arrives via deferred _notify_sent after _on_closed.

Client SSL constructor error handling: The ssl_client constructor handles ssl_ctx.client(host) failure internally — catches the error, sets a flag, and lets _finish_initialization detect it and deliver _on_connection_failure() asynchronously. The user always gets a TCPConnection object and learns about failures through _on_connection_failure(), same as TCP-level connection failures. The ssl_server constructor is partial, matching the existing _on_accept pattern where the listener closes the fd on error.

_connected and _ssl_ready are separate flags: _connected remains TCP-level state (set when the TCP connection succeeds). _ssl_ready is a new flag for SSL handshake completion. User-facing methods (send(), is_writeable()) check both. Internal SSL methods use _send_final() which only checks is_open() (TCP-level), so handshake data (ClientHello, etc.) flows normally. This is the same pattern the interceptor uses today (_connected + _interceptor_ready), with different field names.

is_writeable() semantics: No change needed. During SSL handshake, _connected is true but _ssl_ready is false, so is_writeable() returns false. Before TCP connection, _connected is false.

Post-close send behavior: send() returns SendErrorNotConnected after close. This is the existing behavior (is_open() returns false), unchanged.

SSL-specific callbacks: NetSSLLifecycleEventReceiver (on_alpn_negotiated, on_ssl_auth_failed, on_ssl_error) are intentionally dropped. These can be added back later in an additive fashion. SSL handshake failure maps to _on_connection_failure() for clients, _on_closed() for servers.

Revised implementation steps

Step 1: Add _on_send_failed to lifecycle receiver traits

File: lifecycle_event_receiver.pony

Add to both ServerLifecycleEventReceiver and ClientLifecycleEventReceiver with a docstring following the _on_sent pattern:

fun ref _on_send_failed(token: SendToken) => None

Additive and backwards compatible.

Step 2: Add _notify_send_failed behavior and dispatch method

File: tcp_connection_actor.pony — add behavior:

be _notify_send_failed(token: SendToken) =>
  _connection()._fire_on_send_failed(token)

File: tcp_connection.pony — add dispatch method:

fun ref _fire_on_send_failed(token: SendToken) =>
  match _lifecycle_event_receiver
  | let s: EitherLifecycleEventReceiver ref =>
    s._on_send_failed(token)
  | None =>
    _Unreachable()
  end

Step 3: Modify hard_close() to fire _on_send_failed for pending token

File: tcp_connection.pony

Before clearing _pending and _pending_token, fire _notify_send_failed if a token exists:

match _pending_token
| let t: SendToken =>
  _pending_token = None
  match _enclosing
  | let e: TCPConnectionActor ref =>
    e._notify_send_failed(t)
  | None =>
    _Unreachable()
  end
end

_pending.clear()

Note: The Windows IOCP _write_completed path has a pre-existing TODO about write failure reporting and doesn't use the send token system. Windows-side _on_send_failed integration will need attention when the IOCP token tracking is implemented.

Step 4: Add SSL constructors to TCPConnection

File: tcp_connection.pony

New fields:

var _ssl: (SSL ref | None) = None
var _ssl_ready: Bool = false
var _ssl_failed: Bool = false

none() constructor initializes to defaults (None, false, false).

ssl_server (partial): Takes SSLContext val. Internally calls ssl_ctx.server()? to create the SSL iso session. Otherwise identical to server(). If ssl_ctx.server() raises, the constructor raises — the listener handles this by closing the fd.

ssl_client (non-partial): Takes SSLContext val. Internally tries ssl_ctx.client(host)?. On failure, sets _ssl_failed = true. _finish_initialization detects the flag and delivers _on_connection_failure() asynchronously instead of starting TCP connection.

Both constructors should have docstrings.

Step 5: Add internal SSL logic to TCPConnection

File: tcp_connection.pony

Move logic from SSLClientInterceptor/SSLServerInterceptor into TCPConnection private methods.

On connection established:

• For clients: SSL handshake initiation happens in _event_notify where _connected is set to true (around the current _on_connected() / on_setup() call site). When _ssl is not None, initiate the handshake — flush initial SSL data (ClientHello) via _ssl.can_send() / _ssl.send() → _send_final(). Do NOT deliver _on_connected — wait for handshake completion.
• For servers: SSL handshake initiation happens in _complete_server_initialization (same location as the current on_setup() call). _connected is set to true (needed for _send_final to work). Flush any initial SSL data. Do NOT deliver _on_started — wait for handshake. _queue_read() is called to start receiving the ClientHello from the client.

Incoming data (in _deliver_received):

• If _ssl is not None: feed data to _ssl.receive(), then call _ssl_poll()
• _ssl_poll() checks _ssl.state():
  • SSLReady and not yet _ssl_ready: set _ssl_ready = true, deliver _on_connected / _on_started
  • SSLAuthFail / SSLError: call hard_close(), which triggers _on_connection_failure (for clients pre-ready) or _on_closed (for servers or post-ready)
• Read decrypted data via _ssl.read() loop, deliver to lifecycle receiver
• Flush SSL protocol data via _ssl.can_send() / _ssl.send() → _send_final()

Outgoing data (in send()):

• Add _ssl_ready check: if _ssl is not None and not _ssl_ready, return SendErrorNotConnected
• If _ssl is not None and _ssl_ready: encrypt via _ssl.write(), then send via _ssl.can_send() / _ssl.send() → _send_final()

is_writeable():

• If _ssl is not None: also check _ssl_ready

On close (in hard_close()):

• If _ssl is not None: call _ssl.dispose()
• If _ssl is not None and not _ssl_ready: for clients deliver _on_connection_failure(), for servers deliver _on_closed()

Expect handling:

• If _ssl is not None: store application's expect value internally, tell TCP read layer to use 0. Apply stored value when chunking decrypted data from _ssl.read().
• When removing the interceptor in Step 6, update expect() to use the internal SSL expect logic instead of calling _interceptor.adjust_expect().

Step 6: Remove DataInterceptor infrastructure

Files: tcp_connection.pony, data_interceptor.pony

• Delete data_interceptor.pony entirely (removes DataInterceptor, WireSender, IncomingDataReceiver, InterceptorControl)
• Remove adapter classes from tcp_connection.pony: _WireSenderAdapter, _IncomingDataReceiverAdapter, _InterceptorControlAdapter
• Remove the interceptor parameter from client() and server() constructors
• Remove all interceptor-related fields: _interceptor, _interceptor_ready, _wire_sender, _incoming_receiver, _interceptor_control
• Remove _interceptor_signal_ready() method
• Remove all interceptor branching from send(), _deliver_received(), is_writeable(), hard_close(), expect(), and initialization methods
• Update expect() to use the internal SSL expect handling (check _ssl instead of _interceptor)

Step 7: Remove SSL interceptor classes and SendErrorNotReady

File: net_ssl_connection.pony — delete entirely (removes NetSSLLifecycleEventReceiver, SSLClientInterceptor, SSLServerInterceptor).

File: send_token.pony — remove SendErrorNotReady. Update SendError to (SendErrorNotConnected | SendErrorNotWriteable).

SendErrorNotReady is no longer needed: with SSL built into TCPConnection, _on_connected/_on_started doesn't fire until handshake completes, and send() returns SendErrorNotConnected during handshake (via _ssl_ready check).

Step 8: Update examples

Replace interceptor usage with ssl_client/ssl_server constructors. Non-SSL examples unchanged.

Step 9: Update tests

Build and run tests with make ssl=3.0.x.

• Update _TestSSLPingPong and related actors to use ssl_client/ssl_server constructors
• Add test for _on_send_failed: verify it fires with the correct token when connection drops with pending data
• _TestSendAfterClose — no change needed (still checks for SendErrorNotConnected)
• Remove interceptor-specific test code

Step 10: Update CLAUDE.md

Update to reflect: new constructors, no-queueing design principle, _on_send_failed callback, removal of DataInterceptor and related infrastructure, removal of SendErrorNotReady, updated architecture and examples.

Revised phasing

• Phase 1 (Steps 1-3): Send failure notification. Additive, non-breaking. Small standalone PR.
• Phase 2 (Steps 4-5): SSL constructors and internal SSL logic. Adds new constructors alongside existing interceptor approach.
• Phase 3 (Steps 6-9): Remove old infrastructure, SendErrorNotReady, update examples and tests. Breaking change.
• Phase 4 (Step 10): Documentation.

What changed from the original plan

• Removed: Pre-connection send queue (original Step 2). TCPConnection does not queue application data.
• Removed: Removal of SendErrorNotConnected (original Step 4). This error stays — it covers both not-yet-connected and already-closed.
• Removed: SendErrorNotReady removal moved from Phase 1 to Phase 3 (tied to interceptor removal).
• Added: _on_send_failed fires only for accepted-then-failed sends, not queued sends.
• Corrected: _connected and _ssl_ready are separate flags. _connected is TCP-level; _ssl_ready gates user-facing operations.
• Resolved: Client SSL constructor uses internal failure flag (non-partial). Server SSL constructor is partial.

[SeanTAllen]

Review findings and resolved decisions

Review of the revised plan (comment above) surfaced the following. Design decisions have been resolved through discussion; editorial items are noted for implementation.

Design decisions resolved

ssl_server constructor is non-partial (like ssl_client). The original plan said ssl_server is partial, but this doesn't work: TCPConnection.ssl_server() is called from inside a Pony actor constructor, and actor constructors can't be partial. Instead, ssl_server handles ssl_ctx.server() failure internally using the same _ssl_failed flag pattern as ssl_client, and reports failure asynchronously through a lifecycle event.

New _on_start_failure() callback on ServerLifecycleEventReceiver. Servers need an equivalent to _on_connection_failure() for clients. Any server failure before _on_started would have fired (both ssl_ctx.server() construction failure and SSL handshake failure) delivers _on_start_failure(). Default implementation is None. This replaces the plan's earlier statement that server SSL failures deliver _on_closed().

Editorial corrections for implementation

• The design decisions section says "No change needed" for is_writeable() — should say "No change to semantics." Steps 5 and 6 correctly describe the implementation changes needed.
• Step 3's insertion makes the existing _pending_token = None in hard_close() (line 234) redundant. Remove it during implementation.
• The internal field for storing the application's expect value when SSL is active should be named in the plan (e.g., _ssl_expect).
• The expect() update instruction appears in both Step 5 and Step 6. It's a Step 6 action (when the interceptor is removed).
• The _on_send_failed docstring should document the ordering: it fires in a subsequent turn after _on_closed, consistent with _on_sent.
• During Phase 2, the new SSL path returns SendErrorNotConnected during handshake while the old interceptor path returns SendErrorNotReady. This is a temporary inconsistency — no single user sees both paths — and resolves when SendErrorNotReady is removed in Phase 3.
• Phase 2 coexistence: _ssl and _interceptor must never both be non-None simultaneously. This invariant should be documented during implementation.

[SeanTAllen]

The revised plan was fully implemented across two PRs:

• PR Add _on_send_failed and _on_start_failure lifecycle callbacks #159 (Phase 1) — _on_send_failed, _on_start_failure, _notify_send_failed
• PR Make SSL a first-class TCPConnection concern #160 (Phases 2–4) — SSL constructors, internal SSL logic, DataInterceptor removal, example/test/doc updates

Send queueing (Steps 2 and 4 of the original plan) was dropped during implementation — send() stays fallible instead of queueing.