API Proposal: Public TLS Handles Bound to a Socket — `SafeTlsContextHandle`, `SafeTlsHandle`

[DeagleGross]

TLDR - show me the code

Here is the vibe-coded implementation
Note: this is only the rough implementation, it will be redone carefully once API is approved

Background and motivation

SslStream is the only way to do TLS on .NET today. On Linux it sits on top of OpenSSL via memory BIOs (bio_s_mem). That architecture forces two extra memcpy operations on every TLS read and every TLS write:

Read path  : kernel → managed buffer → BIO_write → OpenSSL → SSL_read out → managed buffer
Write path : managed → SSL_write → OpenSSL → BIO_read → managed → kernel

The alternative is to bind the TLS connection directly to a socket file descriptor, so the TLS provider recv/sends into the kernel itself with no application-side copy. On OpenSSL this is the SSL_set_fd model.

Today there is no public managed surface for this. The OpenSSL P/Invoke layer (Interop.Ssl, Interop.SslCtx) is internal to System.Net.Security. Every server author who wants the direct-fd model has to re-declare the entire OpenSSL P/Invoke surface in their own repo.

This proposal adds two public SafeHandle types in System.Net.Security that represent a TLS configuration context and a TLS connection bound to a socket. Operations and negotiated-info getters live as instance methods/properties on the handles. Configuration uses the existing SslServerAuthenticationOptions / SslClientAuthenticationOptions types.

This proposal does not introduce any state machine, polling loop, async wrapper, or per-connection orchestration.

Naming

• All new types use Tls prefix (not Ssl) — consistent with modern .NET naming (e.g., TlsCipherSuite). Avoids confusion with the existing internal Microsoft.Win32.SafeHandles.SafeSslHandle.
• Names are provider-opaque: SafeTlsContextHandle and SafeTlsHandle describe what the types are, not how they are implemented underneath. This mirrors SslStream itself.
• New types live in System.Net.Security (the feature namespace, where SslStream lives).
• Initial implementation is OpenSSL-on-Linux, gated by [UnsupportedOSPlatform].Other providers may be supported later.

What each handle represents

SafeTlsContextHandle — long-lived TLS configuration context

The TLS configuration object: certificate(s), private key, protocol version range, ALPN protocol list, session cache parameters. Create one per listener, share across all connections that listener accepts. Thread-safe. Released on Dispose.

Internally wraps OpenSSL's SSL_CTX* (via SSL_CTX_new / SSL_CTX_free). Configuration is applied from SslServerAuthenticationOptions or SslClientAuthenticationOptions during factory construction — no individual setup methods are exposed.

SafeTlsHandle — per-connection TLS session bound to a socket

The per-connection TLS state, bound to a socket file descriptor (no memory BIOs). Owns the handshake state, the negotiated session info (protocol version, cipher suite, ALPN, peer cert, SNI), and a back-reference to its SafeTlsContextHandle. Released on Dispose.

Internally wraps OpenSSL's SSL* (via SSL_new → SSL_set_fd → SSL_free).

Note on SSL_set_fd. This is the one piece of native plumbing that doesn't exist in dotnet/runtime today — because SslStream always uses memory BIOs (SSL_set_bio). A new internal Interop.Ssl.SslSetFd P/Invoke is needed (one-line addition). Every other native call is reused as-is.

The factory takes a SafeSocketHandle and DangerousAddRefs it for the lifetime of the handle, so the fd cannot be invalidated under the TLS provider.

Goals

• Public SafeHandle-typed wrappers for the TLS context and TLS connection, bound to a socket fd (no memory BIOs).
• Small set of operations to drive a non-blocking TLS connection end-to-end.
• Configuration via existing SslServerAuthenticationOptions / SslClientAuthenticationOptions — no new configuration API to learn.
• Reuse existing exception types (AuthenticationException, IOException) — no new exception types.
• Guardrails: Read/Write before handshake completion throws InvalidOperationException.

API Proposal

namespace System.Net.Security;

/// <summary>
/// Status returned from non-blocking TLS operations on <see cref="SafeTlsHandle"/>.
/// </summary>
/// <remarks>
/// Maps to OpenSSL's <c>SSL_get_error()</c> classification in provider-opaque terms.
/// Schannel's <c>SEC_I_CONTINUE_NEEDED</c> / <c>SEC_E_INCOMPLETE_MESSAGE</c> map
/// onto the same enum values if a Windows backing is added later.
/// </remarks>
public enum TlsOperationStatus
{
    /// <summary>
    /// Operation completed successfully. For <see cref="SafeTlsHandle.Read"/>:
    /// <c>bytesRead</c> bytes were produced, or <c>bytesRead == 0</c> means
    /// the peer sent <c>close_notify</c> (clean shutdown).
    /// </summary>
    Complete = 0,

    /// <summary>
    /// The TLS provider needs to read from the underlying socket before it can
    /// make progress. Wait for socket-readable, then call the same method again.
    /// </summary>
    WantRead = 1,

    /// <summary>
    /// The TLS provider needs to write to the underlying socket before it can
    /// make progress. Wait for socket-writable, then call the same method again.
    /// </summary>
    WantWrite = 2,

    /// <summary>
    /// The underlying transport is gone (RST, unexpected EOF before <c>close_notify</c>).
    /// The caller should dispose the <see cref="SafeTlsHandle"/>.
    /// </summary>
    Closed = 3,
}

namespace System.Net.Security;

/// <summary>
/// A long-lived TLS configuration context. Thread-safe; create once per listener,
/// share across many <see cref="SafeTlsHandle"/> instances.
/// </summary>
/// <remarks>
/// Internally wraps OpenSSL's <c>SSL_CTX*</c>. Configuration is applied from
/// <see cref="SslServerAuthenticationOptions"/> or <see cref="SslClientAuthenticationOptions"/>
/// during factory construction.
/// </remarks>
[UnsupportedOSPlatform("windows")]
[UnsupportedOSPlatform("browser")]
[UnsupportedOSPlatform("wasi")]
public sealed class SafeTlsContextHandle : SafeHandle
{
    public override bool IsInvalid { get; }
    protected override bool ReleaseHandle();

    /// <summary>
    /// Creates a server-side TLS context configured from the given options.
    /// </summary>
    /// <param name="options">
    /// Server authentication options. Properties mapped to native configuration:
    /// <see cref="SslServerAuthenticationOptions.ServerCertificate"/> or
    /// <see cref="SslServerAuthenticationOptions.ServerCertificateContext"/> → certificate/key,
    /// <see cref="SslServerAuthenticationOptions.EnabledSslProtocols"/> → protocol range,
    /// <see cref="SslServerAuthenticationOptions.ApplicationProtocols"/> → ALPN,
    /// <see cref="SslServerAuthenticationOptions.AllowRenegotiation"/> → renegotiation policy,
    /// <see cref="SslServerAuthenticationOptions.AllowTlsResume"/> → session cache,
    /// <see cref="SslServerAuthenticationOptions.CipherSuitesPolicy"/> → cipher suites.
    /// </param>
    /// <exception cref="AuthenticationException">
    /// Certificate or key configuration failed (e.g., private key mismatch).
    /// </exception>
    /// <exception cref="PlatformNotSupportedException">
    /// The current platform does not support direct-fd TLS.
    /// </exception>
    public static SafeTlsContextHandle Create(SslServerAuthenticationOptions options);

    /// <summary>
    /// Creates a client-side TLS context configured from the given options.
    /// </summary>
    /// <exception cref="PlatformNotSupportedException">
    /// The current platform does not support direct-fd TLS.
    /// </exception>
    public static SafeTlsContextHandle Create(SslClientAuthenticationOptions options);

    /// <summary>
    /// Sets the TLS session cache size. Controls how many resumable sessions
    /// are kept in memory. Only meaningful when
    /// <see cref="SslServerAuthenticationOptions.AllowTlsResume"/> is <see langword="true"/>.
    /// </summary>
    /// <remarks>
    /// This is not available on <see cref="SslServerAuthenticationOptions"/> today.
    /// It maps to OpenSSL's <c>SSL_CTX_sess_set_cache_size</c>.
    /// </remarks>
    public void SetSessionCacheSize(int size);

    /// <summary>
    /// Sets the TLS session timeout. Sessions older than this are not resumed.
    /// </summary>
    /// <remarks>
    /// This is not available on <see cref="SslServerAuthenticationOptions"/> today.
    /// It maps to OpenSSL's <c>SSL_CTX_set_timeout</c>.
    /// </remarks>
    public void SetSessionTimeout(TimeSpan timeout);
}

namespace System.Net.Security;

/// <summary>
/// A per-connection TLS session bound to a socket file descriptor.
/// All operations are non-blocking — the underlying socket must be non-blocking.
/// </summary>
/// <remarks>
/// <para>
/// Internally wraps OpenSSL's <c>SSL*</c>, bound to the socket via
/// <c>SSL_set_fd</c> (no memory BIOs). The <see cref="SafeSocketHandle"/> is
/// ref-counted via <c>DangerousAddRef</c> for the lifetime of this handle.
/// </para>
/// <para>
/// <see cref="Read"/> and <see cref="Write"/> throw
/// <see cref="InvalidOperationException"/> if the handshake has not completed.
/// The caller must drive <see cref="Handshake"/> to completion first.
/// </para>
/// </remarks>
[UnsupportedOSPlatform("windows")]
[UnsupportedOSPlatform("browser")]
[UnsupportedOSPlatform("wasi")]
public sealed class SafeTlsHandle : SafeHandle
{
    public override bool IsInvalid { get; }
    protected override bool ReleaseHandle();

    /// <summary>
    /// Creates a TLS connection bound to a socket.
    /// </summary>
    /// <param name="context">The TLS configuration context.</param>
    /// <param name="socket">
    /// The socket to bind. Must be non-blocking. The handle is ref-counted
    /// for the lifetime of this <see cref="SafeTlsHandle"/>.
    /// </param>
    /// <param name="isServer">
    /// <see langword="true"/> for server-side (accept state);
    /// <see langword="false"/> for client-side (connect state).
    /// </param>
    /// <exception cref="IOException">
    /// The native <c>SSL_new</c> or <c>SSL_set_fd</c> call failed.
    /// </exception>
    public static SafeTlsHandle Create(
        SafeTlsContextHandle context,
        SafeSocketHandle socket,
        bool isServer);

    // ── Pre-handshake configuration ───────────────────────────────────────

    /// <summary>
    /// Sets the target host name for SNI (client-side) or for the server to
    /// log/match against.
    /// </summary>
    public void SetTargetHostName(string targetHost);

    /// <summary>
    /// Enables quiet shutdown — <see cref="Shutdown"/> will not wait for the
    /// peer's <c>close_notify</c> before returning <see cref="TlsOperationStatus.Complete"/>.
    /// </summary>
    public void SetQuietShutdown(bool enabled);

    // ── Non-blocking operations ───────────────────────────────────────────

    /// <summary>
    /// Drives the TLS handshake forward. Call repeatedly, observing the
    /// returned status and waiting for socket readiness between calls.
    /// </summary>
    /// <returns>
    /// <see cref="TlsOperationStatus.Complete"/> when the handshake finishes.
    /// <see cref="TlsOperationStatus.WantRead"/> or
    /// <see cref="TlsOperationStatus.WantWrite"/> when the socket isn't ready.
    /// <see cref="TlsOperationStatus.Closed"/> if the transport was lost.
    /// </returns>
    /// <exception cref="AuthenticationException">
    /// A TLS protocol error, certificate verification failure, or other
    /// unrecoverable handshake error occurred.
    /// </exception>
    public TlsOperationStatus Handshake();

    /// <summary>
    /// Reads decrypted data from the TLS connection.
    /// </summary>
    /// <exception cref="InvalidOperationException">
    /// The TLS handshake has not completed.
    /// </exception>
    /// <exception cref="IOException">
    /// An unrecoverable I/O error occurred during decryption.
    /// </exception>
    public TlsOperationStatus Read(Span<byte> buffer, out int bytesRead);

    /// <summary>
    /// Writes data to the TLS connection (encrypts and sends).
    /// </summary>
    /// <exception cref="InvalidOperationException">
    /// The TLS handshake has not completed.
    /// </exception>
    /// <exception cref="IOException">
    /// An unrecoverable I/O error occurred during encryption.
    /// </exception>
    public TlsOperationStatus Write(ReadOnlySpan<byte> buffer, out int bytesWritten);

    /// <summary>
    /// Initiates a TLS shutdown (<c>close_notify</c>).
    /// </summary>
    /// <exception cref="IOException">
    /// An unrecoverable I/O error occurred during shutdown.
    /// </exception>
    public TlsOperationStatus Shutdown();

    // ── Negotiated info (valid once Handshake returned Complete) ──────────

    /// <summary>Gets the negotiated TLS protocol version.</summary>
    public SslProtocols NegotiatedProtocol { get; }

    /// <summary>Gets the negotiated cipher suite.</summary>
    public TlsCipherSuite NegotiatedCipherSuite { get; }

    /// <summary>Gets the negotiated ALPN protocol.</summary>
    public SslApplicationProtocol NegotiatedApplicationProtocol { get; }

    /// <summary>Gets the target host name (SNI) if set.</summary>
    public string? TargetHostName { get; }

    /// <summary>
    /// Gets whether this connection resumed a previous TLS session
    /// (session ticket / session ID reuse).
    /// </summary>
    public bool SessionResumed { get; }

    /// <summary>Gets the peer's certificate, if one was presented.</summary>
    public X509Certificate2? GetRemoteCertificate();
}

Error model

Scenario	What happens
WantRead / WantWrite	Returned as TlsOperationStatus. Never throws. Go back to poll, retry.
Clean EOF (close_notify)	Read returns Complete with bytesRead == 0.
Transport gone (RST, unexpected EOF)	Returns TlsOperationStatus.Closed.
Handshake protocol error, cert failure	Handshake throws AuthenticationException — consistent with SslStream.
Post-handshake I/O error	Read/Write throw IOException — consistent with stream I/O conventions.
Read/Write before handshake	Throws InvalidOperationException.
Renegotiation (TLS 1.2)	Transparent — OpenSSL handles it internally. Read/Write return WantRead/WantWrite, consumer retries. No API change needed. Controlled by SslServerAuthenticationOptions.AllowRenegotiation.
Key update (TLS 1.3)	Fully transparent — OpenSSL handles it internally with no visible status change.

Reusing AuthenticationException for handshake failures and IOException for post-handshake errors avoids introducing any new exception types while being consistent with how SslStream behaves today.

API Usage

using System.Net.Security;

// ── One-time per listener ─────────────────────────────────────────────────
var ctx = SafeTlsContextHandle.Create(new SslServerAuthenticationOptions
{
    ServerCertificateContext = SslStreamCertificateContext.Create(myCert, additionalCertificates: null),
    EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
    ApplicationProtocols = [SslApplicationProtocol.Http2, SslApplicationProtocol.Http11],
    AllowTlsResume = true,
    AllowRenegotiation = false,
});
ctx.SetSessionCacheSize(20_480);
ctx.SetSessionTimeout(TimeSpan.FromHours(1));

// ── Per accepted connection ───────────────────────────────────────────────
var tls = SafeTlsHandle.Create(ctx, clientSocketHandle, isServer: true);

// Drive handshake from the consumer's own poll loop:
while (true)
{
    var status = tls.Handshake();
    if (status == TlsOperationStatus.Complete) break;
    if (status == TlsOperationStatus.Closed) { tls.Dispose(); return; }
    WaitForReadiness(clientSocketHandle, status); // consumer's SafePollHandle
}

// Steady-state read (same pattern for write):
Span<byte> buf = stackalloc byte[16 * 1024];
var rs = tls.Read(buf, out int n);
if (rs == TlsOperationStatus.Complete && n > 0)
{
    ProcessData(buf[..n]);
}

Design details

Configuration via SslServerAuthenticationOptions (not individual methods)

The original prototype exposed individual configuration methods on SafeTlsContextHandle: UseCertificate, SetProtocols, SetApplicationProtocols, etc. I replaced this with a factory that accepts SslServerAuthenticationOptions / SslClientAuthenticationOptions:

// Before (individual methods):
var ctx = SafeTlsContextHandle.CreateServer();
ctx.UseCertificateContext(certCtx);
ctx.SetProtocols(SslProtocols.Tls12 | SslProtocols.Tls13);
ctx.SetApplicationProtocols(alpnList);

// After (options-based factory):
var ctx = SafeTlsContextHandle.Create(new SslServerAuthenticationOptions
{
    ServerCertificateContext = certCtx,
    EnabledSslProtocols = SslProtocols.Tls12 | SslProtocols.Tls13,
    ApplicationProtocols = alpnList,
});

• Zero new configuration API — users already know these options from SslStream.AuthenticateAsServerAsync.
• All properties mapped at once — no risk of forgetting to call a setup method.
• Immutable after construction — the context can't be mutated after connections start using it.

Two configuration knobs are not on SslServerAuthenticationOptions today: SessionCacheSize and SessionTimeout. These remain as instance methods on SafeTlsContextHandle until they can be added to the options type.

Guardrails — handshake state tracking

SafeTlsHandle tracks whether the handshake has completed. Read and Write throw InvalidOperationException if called before Handshake returns TlsOperationStatus.Complete:

public TlsOperationStatus Read(Span<byte> buffer, out int bytesRead)
{
    if (!_handshakeCompleted)
        throw new InvalidOperationException("TLS handshake has not been completed.");
    // ... SSL_read ...
}

This prevents subtle bugs where SSL_read called before handshake would return confusing errors. The overhead is a single bool check — negligible even on the hot path.

Renegotiation and TLS 1.3 key update

TLS 1.2 renegotiation: If the peer initiates renegotiation during Read/Write, OpenSSL handles it transparently. SSL_read/SSL_write return SSL_ERROR_WANT_READ or SSL_ERROR_WANT_WRITE while the renegotiation handshake progresses. The consumer's existing retry loop (go back to poll, retry the same call) handles this with no API changes. Whether renegotiation is allowed is controlled by SslServerAuthenticationOptions.AllowRenegotiation, which maps to SSL_OP_NO_RENEGOTIATION internally.

TLS 1.3 key update: TLS 1.3 replaced renegotiation with KeyUpdate messages. OpenSSL handles these entirely internally — the consumer never sees them. No API consideration needed.

TLS 1.3 Post-Handshake Authentication (PHA): If ClientCertificateRequired is set and the negotiated protocol is TLS 1.3, the runtime's existing CryptoNative_SslRenegotiate function handles PHA via SSL_verify_client_post_handshake + SSL_do_handshake. This could be exposed as a future method if needed.

Cross-provider comparison

BoringSSL / LibreSSL: Both are API-compatible with OpenSSL for the calls we use (SSL_CTX_new, SSL_new, SSL_set_fd, SSL_do_handshake, SSL_read, SSL_write, SSL_shutdown). No special conditionals are needed. The runtime's native layer already uses OpenSSL version-based conditionals (OPENSSL_VERSION_*) that cover BoringSSL/LibreSSL transparently.

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/ncl, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[teo-tsirpanis]

Instead of new APIs, would it be possible to optimize SslStream to avoid the memory copies if it wraps a NetworkStream or a PipeStream?

Also, how would this work on non-OpenSSL platforms?

[rzikm]

I am not a fan of introducing APIs which will be working only for a specific platform, especially something as low level as this which is likely to be only used by Kestrel internally.

I would be more open to exposing Stream-less, span-based (or similar) API that would represent just the TLS state machine, that one we could create in a more portable manner (there might be some obstacles on OSX with the new NetworkFramework though) and would still remove some copies from between Pipeliens and OpenSSL path. This might be reasonably easy experiment to do to see if it brings similar benefits.

We could also experiment with custom BIO_METHOD implementation that would rid us of one additional copy, that would benefit the current SslStream implementation as is (linux only).

One small issue with special-casing SslStream-over-Socket/NetworkStream and letting OpenSSL talk directly to the socket is making Async I/O work. I wasn't able to make it work in a way that wouldn't be slower than current implementation, maybe once we have parts of the SocketAsyncEngine available as part of public threadpool API, we could try again.

[DeagleGross]

We could also experiment with custom BIO_METHOD implementation that would rid us of one additional copy, that would benefit the current SslStream implementation as is (linux only).

One small issue with special-casing SslStream-over-Socket/NetworkStream and letting OpenSSL talk directly to the socket is making Async I/O work. I wasn't able to make it work in a way that wouldn't be slower than current implementation, maybe once we have parts of the SocketAsyncEngine available as part of public threadpool API, we could try again.

The intention so far is to avoid calling SslStream completely, and given we have a proof in benchmarks that perf gain can be reached, I think improving SslStream is out of scope in this issue (unless there is a clear idea of what we want to try out).

There's also a Kestrel-specific reason to bypass SslStream that's separate from the memcpy story: SslStream is a Stream, but Kestrel's transport surface is IDuplexPipe. Using TLS today forces a StreamPipeReader/StreamPipeWriter adapter in front of it, which reintroduces buffer copies between Pipe segments and Stream byte arrays, and means TLS connections take a fundamentally different path through Kestrel than plaintext ones. The fd-binding shape lets the TLS connection expose an IDuplexPipe natively — no Stream adapter, no second concurrency model, no extra copy at the adapter boundary.

APIs which will be working only for a specific platform

It will work with any OpenSSL / BoringSSL (or similar crypto providers) under the hood. From my understanding any Unix system with OpenSSL will work here well. Windows has a different TLS provider (Schannel) which has no fd-binding model at all, so it won't be supported. I'm fine marking the API explicitly as [SupportedOSPlatform("linux")] (or similar) and naming it so it's clear at the API level that this is the OpenSSL-direct-fd primitive rather than a generic TLS connection type.

I would be more open to exposing Stream-less, span-based (or similar) API that would represent just the TLS state machine, that one we could create in a more portable manner (there might be some obstacles on OSX with the new NetworkFramework though) and would still remove some copies from between Pipeliens and OpenSSL path. This might be reasonably easy experiment to do to see if it brings similar benefits.

The whole point of this proposal is to let the TLS provider read from the socket directly via fd binding — that's what removes the application from the I/O hot path entirely. A span-based state-machine API can't express that: by construction the consumer has to recv() itself, hand the bytes back to TLS, take plaintext out, and route it onward.

making Async I/O work. I wasn't able to make it work in a way that wouldn't be slower than current implementation

ere are other optimizations that work well when owning the socket, file descriptor and epoll directly — EPOLLEXCLUSIVE accept, TCP_DEFER_ACCEPT, accept4 on the worker thread, syscall + decrypt on the same thread that got the epoll wake.

What I'm not sure how to reach with either SslStream or a TLS state-machine API is that same set of properties without shipping a second epoll loop alongside SocketAsyncEngine. If there's a shape you have in mind that lets the runtime own the polling and still gives the consumer this level of control, I'm happy to prototype it

[rzikm]

I think improving SslStream is out of scope in this issue (unless there is a clear idea of what we want to try out).

Sure, that can be done independently of this API proposal.

Using TLS today forces a StreamPipeReader/StreamPipeWriter adapter in front of it, which reintroduces buffer copies between Pipe segments and Stream byte arrays, and means TLS connections take a fundamentally different path through Kestrel than plaintext ones.

I wouln't say that the difference is fundamental, you usually have also HTTP layer with those pipe transports on top of that, so the only difference between HTTPS and HTTP is one extra layer of pipes (be it with an adapter for Stream).

It will work with any OpenSSL / BoringSSL (or similar crypto providers) under the hood. From my understanding any Unix system with OpenSSL will work here well.

Yes, that's what I meant with "only for a specific platform". This will work on Linux, and maybe some UNIX variants like freebsd. It will not be useful for OSX or Windows, or mobile platforms for that matter. It is difficult to push for an API that will make sense only on one major desktop platform.

ere are other optimizations that work well when owning the socket, file descriptor and epoll directly — EPOLLEXCLUSIVE accept, TCP_DEFER_ACCEPT, accept4 on the worker thread, syscall + decrypt on the same thread that got the epoll wake.

This brings up another important question, to use the new API efficiently, you require users to implement a different threading model than is traditionally used in .NET. i.e. this model is not compatible with traditional async/await model as there is no way to asynchronously wait until incoming data arrives (or I don't see it in the API usage). Or am I missing something?

[bartonjs]

How much of this proposal would go away if we just allowed creating an SslStream more transparently to a socket? (an fd, or some socket opening options, et cetera)

The shape of having a SafeHandle type that has public instance members on it is pretty unusual, at that point you're really just being a TlsSession object.

And, I agree with @rzikm here... whatever API is being proposed needs to work on Windows, as well. It doesn't need all the Stream machinery, but it needs to neither be transparently OpenSSL (we don't support calling BoringSSL or any other OSSL clone) nor "looks just like OpenSSL but doesn't use the name".

[rzikm]

How much of this proposal would go away if we just allowed creating an SslStream more transparently to a socket?

We can special-case SslStream-over-NetworkStream without any API changes, I tried something like that in #125110, but it did not bring any perf improvements (in HttpClient benchmarks). I suspect some perf was lost on how the WIP does async wait until data arrives on socket (so that then OpenSSL can read them synchronously in SSL_read).

[rzikm]

This brings up another important question, to use the new API efficiently, you require users to implement a different threading model than is traditionally used in .NET. i.e. this model is not compatible with traditional async/await model as there is no way to asynchronously wait until incoming data arrives (or I don't see it in the API usage). Or am I missing something?

This might be bridged by #47631.

[DeagleGross]

This brings up another important question, to use the new API efficiently, you require users to implement a different threading model than is traditionally used in .NET. i.e. this model is not compatible with traditional async/await model as there is no way to asynchronously wait until incoming data arrives (or I don't see it in the API usage). Or am I missing something?>

It is users decision on what they want to do with the API. My I/O model does not use runtime thread-pool to manage connections, therefore #47631 will not help me anyhow. async/await model is also unrelated - SslStream also under the hood calls ssl_do_handshake. Since I dont want random thread-pool thread to push the tls handshake processing and performing ssl_read and ssl_write, then I need this API. It is intentionally a lower-level one, and is designed to be used for users who really understand what they are doing (and who want to call OpenSSL directly with basically same API provided there) - others can still continue using SslStream or any other mechanism they want to.

How much of this proposal would go away if we just allowed creating an SslStream more transparently to a socket? (an fd, or some socket opening options, et cetera)

The one major difference I want to do is to manage threads which will be managing connections manually. For example I'll create 4 threads, each owns a set of connections and has its own epoll which gives me a way to listen to events happening on the socket. Then I can manage my thread to do ssl_do_handshake / ssl_read / ssl_write depending on state of the connection.

SslStream providers the API to pass in the buffer of encrypted data, and get the un-encrypted data back (because it is a stream). I dont think that expanding options on SslStream gives me same full control I expect, and thread-pool threads will still continue doing the actual interop calls without epoll synchronization. I dont have a clear idea on how to make it work with SslStream.

The shape of having a SafeHandle type that has public instance members on it is pretty unusual, at that point you're really just being a TlsSession object.

Let me try to see what I can do with refactoring to some "TLS"-centric place. I'll try to see what I can do making API be supported on Windows as well. Thanks a lot for your comments and suggestions Jeremy and Radek, I'll post an alternative ASAP.

[DeagleGross]

@bartonjs @rzikm since you have proposed to refactor API to more generic TlsSession - I redesigned the API. Here is a sample if you want to take a look at that in action: DeagleGross/Baraholka#4

Please let me know what you think!

Shape

TlsContext                              ← cross-platform configuration
TlsSession (abstract)                   ← shared lifecycle + negotiated info + factories
  ├─ TlsDetachedSession                 ← caller owns I/O
  │                                       ProcessHandshake / Decrypt / Encrypt
  │                                       Cross-platform
  └─ TlsSocketBoundSession              ← session owns the socket transport
                                          Handshake / Read / Write
                                          [SupportedOSPlatform("linux")]

TlsOperationStatus { Complete, WantRead, WantWrite, Closed }

The two derivations share everything that isn't an I/O method: handshake-complete tracking, SNI / target host, negotiated protocol / cipher / ALPN, session-resumed flag, peer certificate, Shutdown, Dispose. They differ in who owns the transport resource:

Type	I/O driver	Why it exists
TlsDetachedSession	the caller (passes ciphertext/plaintext spans in and out). Analogue of SslStream	Portable shape every TLS provider can implement.
TlsSocketBoundSession	the session (TLS provider performs recv/send itself on a bound fd)	Lets consumers avoid the Stream↔Pipe adapter, keep the syscall + decrypt on one thread, and avoid copying ciphertext through the application

Platform support

I do not have big knowledge of Schannel / Network.framework, but seems like nobody except OpenSSL/BoringSSL supports file-descriptor API (TlsSocketBoundSession). TlsDetachedSession may be supported by all of them. Anyway with this API there will be a way to use the API on any platform (via TlsSession).

API surface

All APIs below are new.

namespace System.Net.Security;

/// <summary>
/// Outcome of a non-blocking TLS operation. Provider-opaque — does not leak OpenSSL
/// error codes or Schannel SECURITY_STATUS values.
/// </summary>
public enum TlsOperationStatus
{
    /// <summary>
    /// Operation completed successfully. For <see cref="TlsDetachedSession.Decrypt"/>,
    /// <c>produced == 0</c> indicates the peer sent <c>close_notify</c> (clean shutdown).
    /// </summary>
    Complete = 0,

    /// <summary>
    /// The TLS layer needs more bytes from the peer before it can make progress.
    /// </summary>
    WantRead = 1,

    /// <summary>
    /// The TLS layer needs to send bytes to the peer before it can make progress.
    /// </summary>
    WantWrite = 2,

    /// <summary>
    /// The transport is gone (RST, unexpected EOF before <c>close_notify</c>).
    /// The caller should dispose the session.
    /// </summary>
    Closed = 3,
}

/// <summary>
/// A long-lived TLS configuration context. Thread-safe; create once per listener (server)
/// or per logical client, then share across many <see cref="TlsSession"/> instances.
/// </summary>
public sealed class TlsContext : IDisposable
{
    /// <summary>Create a server context from existing authentication options.</summary>
    public static TlsContext Create(SslServerAuthenticationOptions options);

    /// <summary>Create a client context from existing authentication options.</summary>
    public static TlsContext Create(SslClientAuthenticationOptions options);

    public void Dispose();
}

/// <summary>
/// A TLS session. Concrete derivations differ in how they own the transport:
/// <see cref="TlsDetachedSession"/> (caller-driven I/O, cross-platform) or
/// <see cref="TlsSocketBoundSession"/> (session-driven I/O on a bound socket; Linux only).
/// </summary>
public abstract class TlsSession : IDisposable
{
    private protected TlsSession();

    /// <summary>True once the TLS handshake has completed.</summary>
    public bool IsHandshakeComplete { get; }

    /// <summary>SNI / target host name. Set on a client session before the first handshake call.</summary>
    public string? TargetHostName { get; set; }

    // ── Negotiated info (valid after IsHandshakeComplete == true) ─────────────
    public SslProtocols NegotiatedProtocol { get; }
    public TlsCipherSuite NegotiatedCipherSuite { get; }
    public SslApplicationProtocol NegotiatedApplicationProtocol { get; }
    public bool SessionResumed { get; }
    public X509Certificate2? GetRemoteCertificate();

    /// <summary>Drives an orderly TLS shutdown (close_notify).</summary>
    public abstract TlsOperationStatus Shutdown();

    /// <summary>
    /// Creates a detached session. The caller drives I/O and uses
    /// <see cref="TlsDetachedSession.ProcessHandshake"/> / <see cref="TlsDetachedSession.Decrypt"/> /
    /// <see cref="TlsDetachedSession.Encrypt"/>.
    /// </summary>
    public static TlsDetachedSession CreateDetached(TlsContext context, bool isServer);

    /// <summary>
    /// Creates a socket-bound session (Linux / OpenSSL only). The session owns the socket and
    /// performs reads / writes on it; the caller drives the state machine via
    /// <see cref="TlsSocketBoundSession.Handshake"/> / <see cref="TlsSocketBoundSession.Read"/> /
    /// <see cref="TlsSocketBoundSession.Write"/>.
    /// </summary>
    [SupportedOSPlatform("linux")]
    public static TlsSocketBoundSession CreateSocketBound(
        TlsContext context,
        SafeSocketHandle socket,
        bool isServer);

    public void Dispose();
}

/// <summary>
/// A TLS session whose ciphertext I/O is driven entirely by the caller through byte spans.
/// Cross-platform: implementable on any TLS provider that exposes a buffer-in / buffer-out
/// surface (OpenSSL memory BIOs, Schannel <c>EncryptMessage</c>/<c>DecryptMessage</c>,
/// Apple SecureTransport callbacks).
/// </summary>
public sealed class TlsDetachedSession : TlsSession
{
    /// <summary>
    /// Drives one step of the handshake. Feed any ciphertext received from the peer in
    /// <paramref name="input"/>; any ciphertext the TLS layer wants to send is written
    /// into <paramref name="output"/>.
    /// </summary>
    /// <returns>
    /// <see cref="TlsOperationStatus.Complete"/> when the handshake is finished.
    /// <see cref="TlsOperationStatus.WantRead"/> when more input is needed from the peer.
    /// <see cref="TlsOperationStatus.WantWrite"/> when there are still bytes pending to send
    /// (call again with a larger output buffer).
    /// </returns>
    public TlsOperationStatus ProcessHandshake(
        ReadOnlySpan<byte> input,
        Span<byte> output,
        out int consumed,
        out int produced);

    /// <summary>Decrypts ciphertext received from the peer into plaintext.</summary>
    public TlsOperationStatus Decrypt(
        ReadOnlySpan<byte> ciphertext,
        Span<byte> plaintext,
        out int consumed,
        out int produced);

    /// <summary>Encrypts plaintext into ciphertext ready to send to the peer.</summary>
    public TlsOperationStatus Encrypt(
        ReadOnlySpan<byte> plaintext,
        Span<byte> ciphertext,
        out int consumed,
        out int produced);

    public override TlsOperationStatus Shutdown();
}

/// <summary>
/// A TLS session that owns its socket transport. The TLS layer performs reads / writes
/// directly on the socket file descriptor; the caller drives the state machine via
/// <see cref="Handshake"/> / <see cref="Read"/> / <see cref="Write"/> and observes
/// <see cref="TlsOperationStatus.WantRead"/> / <see cref="TlsOperationStatus.WantWrite"/>
/// to know when to wait for socket readiness (typically via epoll).
/// </summary>
/// <remarks>
/// Available only on Linux. On platforms whose TLS provider does not expose a socket-bound
/// mode (Schannel, Network.framework), use <see cref="TlsDetachedSession"/> instead.
/// </remarks>
[SupportedOSPlatform("linux")]
public sealed class TlsSocketBoundSession : TlsSession
{
    /// <summary>The socket the session reads from and writes to.</summary>
    public SafeSocketHandle Socket { get; }

    /// <summary>Drives the TLS handshake. Call repeatedly, observing the returned status.</summary>
    public TlsOperationStatus Handshake();

    /// <summary>Reads decrypted application data from the connection.</summary>
    public TlsOperationStatus Read(Span<byte> buffer, out int bytesRead);

    /// <summary>Writes application data through the connection.</summary>
    public TlsOperationStatus Write(ReadOnlySpan<byte> buffer, out int bytesWritten);

    public override TlsOperationStatus Shutdown();
}

Usage

Detached (cross-platform)

using var ctx = TlsContext.Create(new SslServerAuthenticationOptions
{
    ServerCertificateContext = SslStreamCertificateContext.Create(myCert, null),
    EnabledSslProtocols      = SslProtocols.Tls12 | SslProtocols.Tls13,
    ApplicationProtocols     = [SslApplicationProtocol.Http2, SslApplicationProtocol.Http11],
});

using var session = TlsSession.CreateDetached(ctx, isServer: true);

byte[] inBuf  = ArrayPool<byte>.Shared.Rent(16 * 1024);
byte[] outBuf = ArrayPool<byte>.Shared.Rent(16 * 1024);
int inLen = 0;

// Handshake loop — caller owns I/O.
while (!session.IsHandshakeComplete)
{
    var status = session.ProcessHandshake(
        inBuf.AsSpan(0, inLen), outBuf,
        out int consumed, out int produced);

    if (produced > 0) socket.Send(outBuf, 0, produced, SocketFlags.None);

    if (consumed > 0)
    {
        Array.Copy(inBuf, consumed, inBuf, 0, inLen - consumed);
        inLen -= consumed;
    }

    if (status == TlsOperationStatus.WantRead)
    {
        inLen += socket.Receive(inBuf, inLen, inBuf.Length - inLen, SocketFlags.None);
    }
    else if (status == TlsOperationStatus.Closed)
    {
        return;
    }
}

// Steady state — Decrypt for inbound, Encrypt for outbound.
int n = socket.Receive(inBuf);
var ds = session.Decrypt(inBuf.AsSpan(0, n), outBuf, out _, out int produced);
if (ds == TlsOperationStatus.Complete && produced > 0)
{
    ProcessApplicationData(outBuf.AsSpan(0, produced));
}

SocketBound (Linux fast path)

using var ctx = TlsContext.Create(serverOptions);

// Driven from an epoll-based connection pump in this example.
using var session = TlsSession.CreateSocketBound(ctx, clientSocket.SafeHandle, isServer: true);

// Handshake — pump receives socket-readable events and re-drives Handshake().
while (!session.IsHandshakeComplete)
{
    var status = session.Handshake();
    switch (status)
    {
        case TlsOperationStatus.Complete: break;
        case TlsOperationStatus.WantRead:  await pump.WaitReadableAsync(clientSocket); break;
        case TlsOperationStatus.WantWrite: await pump.WaitWritableAsync(clientSocket); break;
        case TlsOperationStatus.Closed:    return;
    }
}

// Steady state — TLS layer talks to the socket itself.
Span<byte> buf = stackalloc byte[16 * 1024];
var rs = session.Read(buf, out int n);
if (rs == TlsOperationStatus.Complete && n > 0) ProcessApplicationData(buf[..n]);