Redesigning Lori's Send System

[SeanTAllen]

Ref: #136

Problem

Lori's send system mirrors the Pony standard library's net package design, which makes choices that don't fit Lori's architecture:

1. Silent failures: send() does not check connection state. Data sent on a closed connection is silently dropped by _send_final. No error is reported to the caller.

2. Library-managed queuing: On POSIX, _send_final pushes data to a List[(ByteSeq, USize)] and drains it via _send_pending_writes. During backpressure the library buffers indefinitely rather than letting the application decide whether to queue, drop, or take other action.

3. No send completion tracking: No mechanism exists to know when data has been handed to the OS. The application can't correlate a send() call with actual transmission.

4. No writeability query: The enclosing actor can't check socket writeability before sending.

Goals

From #136:

1. send becomes fallible — reports failure when the connection can't accept data
2. Send completion tracking — tokens that correlate send() calls with _on_sent() callbacks
3. _on_sent callback — notifies when data has been handed to the OS
4. is_writeable() query — lets the actor check writeability before sending
5. Application-level queuing — the library no longer decides buffering policy; the application does

Design

send() Return Type

fun ref send(data: ByteSeq): (SendToken | SendError)

Returns a SendToken on success, or a SendError explaining the failure.

When send() succeeds: The data has been accepted. It may have been fully written to the OS already (common case), or partially written with the remainder pending (backpressure applied). Either way, _on_sent(token) will fire when the data is fully handed to the OS.

When send() fails: No data was sent, no token is issued, no callback will fire. The application decides what to do — queue the data, drop it, close the connection, etc.

SendToken

class val SendToken is Equatable[SendToken]
  """
  Identifies a send operation. Returned by `send()` on success and delivered
  to `_on_sent()` when the data has been fully handed to the OS.
  """
  let id: USize

  new val _create(id': USize) =>
    id = id'

  fun eq(that: box->SendToken): Bool =>
    id == that.id

  fun ne(that: box->SendToken): Bool =>
    not eq(that)

Managed by TCPConnection via a monotonically increasing counter. Private constructor prevents external creation. Token equality is structural (based on id), scoped per connection — each TCPConnection has its own counter. Applications managing multiple connections should pair tokens with connection identity to avoid ambiguity.

SendError

primitive SendErrorNotConnected
  """
  The connection is not yet established or has already been closed.
  """

primitive SendErrorNotWriteable
  """
  The socket is not writeable. This happens during backpressure (a previous
  send is still pending) or when the socket's send buffer is full.
  Wait for `_on_unthrottled` before retrying.
  """

primitive SendErrorNotReady
  """
  A data interceptor (e.g., SSL) is not ready for application data. This
  typically means a handshake is still in progress.
  Wait for `_on_connected` / `_on_started` before retrying.
  """

type SendError is (SendErrorNotConnected | SendErrorNotWriteable
  | SendErrorNotReady)

Three error types because the failure reasons have different semantics and different recovery signals:

• SendErrorNotConnected — permanent; retry won't help without reconnecting
• SendErrorNotWriteable — transient; wait for _on_unthrottled
• SendErrorNotReady — transient; wait for _on_connected / _on_started

is_writeable() Query

fun ref is_writeable(): Bool =>
  is_open() and _writeable and
    match _interceptor
    | let _: DataInterceptor ref => _interceptor_ready
    | None => true
    end

Returns whether the connection can currently accept a send() call. Checks socket writeability AND whether any interceptor layer is ready. Uses the existing _interceptor_ready field, which is set when the interceptor calls signal_ready() during setup. Connections without an interceptor are always considered ready.

_on_sent Callback

Added to both lifecycle event receiver traits:

// On both ClientLifecycleEventReceiver and ServerLifecycleEventReceiver
fun ref _on_sent(token: SendToken) =>
  """
  Called when data from a successful send() has been fully handed to the
  OS. The token matches the one returned by send().

  Always fires in a subsequent behavior turn, never synchronously during
  send(). This guarantees the caller has received and processed the
  SendToken return value before the callback arrives.
  """
  None

Always deferred: _on_sent is always deferred to a subsequent behavior turn via a new behavior on TCPConnectionActor:

// In TCPConnectionActor
be _notify_sent(token: SendToken) =>
  _connection()._fire_on_sent(token)

This deferral applies in all cases — both immediate completions and pending write flushes. The consistency avoids re-entrancy hazards: if _on_sent fired synchronously during _send_pending_writes (before _release_backpressure), a send() call from within the callback could set new pending data, and the subsequent _release_backpressure() would incorrectly fire. Deferring _on_sent ensures backpressure state is settled before the application can act.

// In TCPConnection
fun ref _fire_on_sent(token: SendToken) =>
  match _lifecycle_event_receiver
  | let s: EitherLifecycleEventReceiver ref =>
    s._on_sent(token)
  | None =>
    _Unreachable()
  end

Connection close: If the connection closes while a send is pending, _on_sent will not fire. The application receives _on_closed and can treat any outstanding tokens as implicitly failed.

Pending Writes

The existing _pending: List[(ByteSeq, USize)] is retained. The unbounded growth problem is solved at the send() level: send() refuses new data when not writeable, so user data can contribute at most one item to the list (a partial write). The list also absorbs small, infrequent protocol data from interceptors (e.g., SSL session tickets, key updates) that may arrive via _send_final during incoming() processing.

New fields for token tracking:

var _next_token_id: USize = 0
var _pending_token: (SendToken | None) = None

Revised send() Flow

fun ref send(data: ByteSeq): (SendToken | SendError) =>
  if not is_open() then
    return SendErrorNotConnected
  end

  if not _writeable then
    return SendErrorNotWriteable
  end

  match _interceptor
  | let _: DataInterceptor ref =>
    if not _interceptor_ready then
      return SendErrorNotReady
    end
  end

  _next_token_id = _next_token_id + 1
  let token = SendToken._create(_next_token_id)

  match _interceptor
  | let i: DataInterceptor ref =>
    match _wire_sender
    | let w: _WireSenderAdapter ref =>
      i.outgoing(data, w)
    | None =>
      _Unreachable()
    end

    // Check if the interceptor closed the connection (e.g., SSL
    // encryption error triggered close()).
    if not is_open() then
      return SendErrorNotConnected
    end
  | None =>
    _send_final(data)
  end

  // Determine when to fire _on_sent
  if not _has_pending_writes() then
    // All data sent to OS immediately; defer _on_sent
    match _enclosing
    | let e: TCPConnectionActor ref =>
      e._notify_sent(token)
    | None =>
      _Unreachable()
    end
  else
    // Partial write; _on_sent fires when pending list drains
    _pending_token = token
  end

  token

The interceptor path uses the existing _WireSenderAdapter, which routes each wire.send() call to _send_final. The interceptor's outgoing() may call wire.send() multiple times (e.g., SSL producing multiple encrypted records); the pending list absorbs these naturally. _send_final calls _send_pending_writes() after each push, draining as much as possible.

Revised _send_pending_writes (POSIX)

fun ref _send_pending_writes() =>
  """
  Send pending write data.
  This is POSIX only.
  """
  ifdef posix then
    while _writeable and _has_pending_writes() do
      try
        let node = _pending.head()?
        (let data, let offset) = node()?

        let len = PonyTCP.send(_event, data, offset)?

        if (len + offset) < data.size() then
          // not all data was sent
          node()? = (data, offset + len)
          _apply_backpressure()
        else
          _pending.shift()?
        end
      else
        // Non-graceful shutdown on error.
        hard_close()
      end
    end

    if not _has_pending_writes() then
      // Release backpressure before deferring _on_sent so the
      // application sees settled backpressure state when the
      // callback fires.
      _release_backpressure()

      match _pending_token
      | let t: SendToken =>
        _pending_token = None
        match _enclosing
        | let e: TCPConnectionActor ref =>
          e._notify_sent(t)
        | None =>
          _Unreachable()
        end
      end
    end
  else
    _Unreachable()
  end

_event_notify calls _send_pending_writes() when the socket becomes writeable.

Graceful Close

close() does not wait for pending writes. Pending writes may partially drain between close() and hard_close() (if the socket becomes writeable in between), and the token fires normally if they happen to complete. Otherwise, on hard_close, the pending token is cleared along with the pending list:

// In hard_close, alongside _pending.clear():
_pending_token = None

The abandoned token means no _on_sent fires; _on_closed is the signal that outstanding tokens are implicitly failed.

SSL Integration

The DataInterceptor architecture handles SSL integration with minimal changes. The key interactions:

Writeability check: TCPConnection.is_writeable() checks _interceptor_ready, which is set when the SSL interceptor calls control.signal_ready() after handshake completion. send() returns SendErrorNotReady before the handshake completes.

Outgoing data: SSL's outgoing() method is unchanged. It encrypts data via the SSL library and pushes encrypted chunks to wire.send(). Since send() blocks calls before the handshake completes (via _interceptor_ready check), the SSL interceptor's pre-handshake buffering (_pending list in SSLClientInterceptor / SSLServerInterceptor) becomes dead code and should be removed during implementation.

Protocol data during incoming(): SSL's _ssl_poll() sends protocol data (handshake responses, session tickets, key updates) via wire.send() during incoming() processing. This routes through _WireSenderAdapter → _send_final() → the pending list. If user data is also pending (partial write), the protocol data is appended to the list and drained together. The token fires when the entire list drains, which may include both user data and protocol data. In practice, protocol data is small and infrequent (a few hundred bytes for session tickets), so this adds negligible delay to token delivery.

Protocol data can cause backpressure: If protocol data sent during incoming() processing causes a partial write, _apply_backpressure sets _writeable = false. If the application calls send() from within the same _on_received callback (a common pattern — e.g., echo servers), it will see SendErrorNotWriteable even though the backpressure was caused by protocol overhead, not application data. This is correct behavior (the socket genuinely isn't writeable), but applications should be prepared for send() to fail during _on_received.

Interceptor errors during send(): If the SSL interceptor encounters an error during outgoing() (e.g., _ssl.write() fails), it calls control.close(), which closes the connection. Back in send(), is_open() returns false and SendErrorNotConnected is returned. Any partially encrypted chunks already pushed to the pending list are cleaned up by hard_close().

Platform Considerations: Windows

The public API (send() return type, SendToken, SendError, is_writeable(), _on_sent) is platform-independent. The IOCP-specific implementation details (how _send_final, _send_pending_writes, and _write_completed work under Windows) are deferred to a separate design effort. The conceptual model maps naturally: send() submits data, _write_completed reports completion, and the token fires when the write is confirmed.

Migration Impact

This is a breaking change to all users of send().

Before

_tcp_connection.send(data)
// Hope it worked

After

match _tcp_connection.send(data)
| let token: SendToken =>
  // Data accepted; _on_sent(token) will fire when handed to OS
  None
| let err: SendErrorNotConnected =>
  // Connection is down
  None
| let err: SendErrorNotWriteable =>
  // Backpressured; queue data or drop it (application decision)
  None
| let err: SendErrorNotReady =>
  // Interceptor handshake not yet complete
  None
end

New Optional Callbacks

Applications that want send completion tracking implement _on_sent:

fun ref _on_sent(token: SendToken) =>
  // Data identified by token has been fully handed to the OS

Applications that don't care can rely on the default no-op implementation.

SSL Users

No changes to the SSL wrapping pattern. The SSL interceptor's internal buffering of pre-handshake sends becomes unnecessary — send() fails explicitly with SendErrorNotReady before the handshake completes. Applications that relied on send-before-handshake working silently need to handle the error (queue the data themselves, or wait for _on_connected/_on_started).

Summary of New/Changed API Surface

Element	Change
TCPConnection.send()	Returns (SendToken | SendError) instead of None
TCPConnection.is_writeable()	New query method
TCPConnection._fire_on_sent(token)	New method
SendToken	New class (Equatable[SendToken])
SendErrorNotConnected	New primitive
SendErrorNotWriteable	New primitive
SendErrorNotReady	New primitive
SendError	New type alias
_on_sent(token)	New callback on both lifecycle receiver traits
_notify_sent(token)	New behavior on TCPConnectionActor
_pending_token	New field on TCPConnection
_next_token_id	New field on TCPConnection
SSL _pending list	Dead code; to be removed

[SeanTAllen]

Review

Major: Design needs updating for the DataInterceptor architecture

The main body of the design references _on_send on lifecycle event receivers and _next_lifecycle_event_receiver() chaining — neither of which exist in the current codebase. The DataInterceptor design (Discussion #149 / PR #151) has landed. The current send() routes through interceptor.outgoing(data, wire), not through _on_send() on lifecycle receivers.

The "Interaction with DataInterceptor Design (#149)" section acknowledges compatibility, but it's a postscript rather than the primary design. Specific impacts:

• send() flow: Should show the interceptor path (interceptor.outgoing(data, wire)) as the primary code path, not s._on_send(data).
• SSL integration section: The _on_send return-value approach doesn't apply. SSL's outgoing() uses the push model — it calls wire.send() potentially multiple times. The design needs a collecting WireSender that buffers all chunks from outgoing() into a single buffer, then send() calls _send_final() once. This is mentioned in one sentence in the compatibility section but needs to be a primary design element with its own code sketch.
• _is_send_ready() placement: The design adds this to lifecycle event receivers with chain delegation. But SSL is now an interceptor, not a lifecycle receiver. TCPConnection already tracks _interceptor_ready (set when signal_ready() is called). The readiness check should use this existing field rather than adding a new method to lifecycle receivers.

is_writeable() simplifies with _interceptor_ready

fun ref is_writeable(): Bool =>
  is_open() and _writeable and
    match _interceptor
    | let _: DataInterceptor ref => _interceptor_ready
    | None => true
    end

No need for _is_send_ready() on lifecycle receivers or interceptors. This also means is_writeable() could be fun box (consistent with is_open()), since it only reads fields.

SendError detection also simplifies

if not is_open() then return SendErrorNotConnected end
if not _writeable then return SendErrorNotWriteable end
match _interceptor
| let _: DataInterceptor ref =>
  if not _interceptor_ready then return SendErrorNotReady end
end

Minor: SendToken should implement Equatable[SendToken]

The design defines fun eq(that: SendToken): Bool but doesn't implement Equatable[SendToken]. Needs the trait for == operator support, plus ne.

Coupling comment needed

The timing coupling between _release_backpressure() and _notify_sent() in _send_pending_write is critical — backpressure must be released before _on_sent fires. The implementation should have a comment on the ordering in _send_pending_write itself, since that's where a future maintainer would change the order.

---

Open Question Decisions

Q1: Should close() wait for pending writes? No. close() proceeds without waiting. _on_closed signals that any outstanding tokens are implicitly failed.

Q2: Should _on_sent fire for immediate sends? Yes, always fire, always deferred via behavior. One consistent success mode — every successful send() returns a token, every token gets exactly one _on_sent (unless _on_closed arrives first). The alternative (only fire for deferred sends) creates two success modes the caller must distinguish, with no clean way to tell which mode a given token is in.

Q3: SSL protocol data pending strategy? Keep the pending list. The unbounded growth problem is already solved by send() refusing new user data when not writeable. The list is naturally bounded: at most one partial user write plus small, infrequent protocol data (session tickets, key updates). The token fires when the list drains. The single-user-send invariant is enforced at the send() level, not at _send_final. This avoids a secondary buffer, avoids SSL managing its own protocol data queue, and minimizes the diff in _send_final / _send_pending_writes.

Q4: Windows IOCP details? Separate design effort. The public API (send() return type, is_writeable(), SendToken, _on_sent) is the same regardless of platform. The IOCP differences are internal to TCPConnection.