S26.02: MbedTlsStream Open() unwinds inner transport and SSL state on failure

[DavidCozens]

Parent epic: #279 (E26: TLS Hardening)
Raised by: CodeRabbit on PR #419 (S08.07). Narrowed to mbedTLS-only following design discussion 2026-05-24; OpenSSL parity moves to S26.03 once this story ships.

Problem

When MbedTlsStream_Open fails after the inner transport's Open() has already succeeded, the adapter returns false without unwinding. The inner transport stays open and the SSL state from ssl_setup / ssl_config_defaults is left allocated. Audit of the path from StreamSender.Send down to the socket found three resources that leak on this path:

1. Transport fd — MbedTlsStream_Open does not call SolidSyslogStream_Close(self->Config.Transport) on the failure tail; the next StreamSender reconnect tick then re-enters PosixTcpStream_Open, which clobbers self->Fd with a fresh socket(...) without closing the prior fd (SolidSyslogPosixTcpStream.c:102).
2. SslContext inner allocations from mbedtls_ssl_setup — only freed by mbedtls_ssl_free inside Close.
3. SslConfig inner allocations from mbedtls_ssl_config_defaults — only freed by mbedtls_ssl_config_free inside Close.

All three close if MbedTlsStream_Open calls its own Close on every failure path after the first allocating operation, since Close already unwinds the SSL state and delegates to the transport (SolidSyslogMbedTlsStream.c:77-84).

Scope

Audit every failure point in MbedTlsStream_Open and pin each one. The failure points are:

• MbedTlsStream_ApplySslConfigDefaults returns false (mbedtls_ssl_config_defaults ≠ 0)
• MbedTlsStream_BindContextToConfig returns false (mbedtls_ssl_setup ≠ 0)
• MbedTlsStream_ConfigureExpectedHostname returns false (mbedtls_ssl_set_hostname ≠ 0)
• MbedTlsStream_PerformHandshake returns false — hard error
• MbedTlsStream_PerformHandshake returns false — budget exhausted

For each, the adapter must:

• Close the inner transport (SolidSyslogStream_Close(self->Config.Transport))
• Free the SslContext (mbedtls_ssl_free)
• Free the SslConfig (mbedtls_ssl_config_free)
• Emit a typed error via SolidSyslog_Error against MbedTlsStreamErrorSource (see error codes below), severity SOLIDSYSLOG_SEVERITY_ERROR
• Return false

A subsequent Open on the same handle must succeed cleanly (recovery contract).

Error codes added to SolidSyslogMbedTlsStreamErrors.h

Messages are protocol-level (what the negotiation failed to achieve), not C-call-level (which mbedTLS function returned non-zero).

Code	Triggered by	Message
MBEDTLSSTREAM_ERROR_DEFAULTS_NOT_APPLIED	mbedtls_ssl_config_defaults ≠ 0	"TLS client could not be configured with its default protocol settings; connection not established"
MBEDTLSSTREAM_ERROR_SESSION_INIT_FAILED	mbedtls_ssl_setup ≠ 0	"TLS session could not be initialised against the configured context; connection not established"
MBEDTLSSTREAM_ERROR_SERVER_NAME_NOT_SET	mbedtls_ssl_set_hostname ≠ 0	"TLS server name (SNI / cert match) could not be configured; connection not established"
MBEDTLSSTREAM_ERROR_HANDSHAKE_REJECTED	Handshake returns a hard error (peer rejected, cert untrusted, protocol incompatible)	"TLS handshake rejected by peer or local verification; connection not established"
MBEDTLSSTREAM_ERROR_HANDSHAKE_TIMEOUT	Handshake retry budget exhausted	"TLS handshake did not complete within the bounded retry budget; connection not established"

Inner-transport Open failure is not a MbedTlsStream error — the transport owns its own diagnostics.

Acceptance criteria

• MbedTlsStream_Open calls an internal unwind (either MbedTlsStream_Close or an equivalent helper) on every failure path after the first allocating operation.
• Five new codes added to SolidSyslogMbedTlsStreamErrors.h (see table above), with messages in SolidSyslogMbedTlsStreamMessages.c. Each failure point emits the matching code via SolidSyslog_Error(SOLIDSYSLOG_SEVERITY_ERROR, &MbedTlsStreamErrorSource, <code>) before returning.
• MbedTlsFake gains return-value setters for mbedtls_ssl_config_defaults, mbedtls_ssl_setup, and mbedtls_ssl_set_hostname (only handshake / read / write are currently settable).
• Six new / extended unit tests in Tests/MbedTls/SolidSyslogMbedTlsStreamTest.cpp pin the contract. Each of tests 1–5 asserts: Open returned false, StreamFake_CloseCallCount(transport) == 1, MbedTlsFake_SslFreeCallCount() == 1, MbedTlsFake_SslConfigFreeCallCount() == 1, ErrorHandlerFake_Handle called once with MbedTlsStreamErrorSource, the matching code, and severity SOLIDSYSLOG_SEVERITY_ERROR.
  1. OpenClosesTransportAndFreesSslStateWhenSslConfigDefaultsFails
  2. OpenClosesTransportAndFreesSslStateWhenSslSetupFails
  3. OpenClosesTransportAndFreesSslStateWhenSetHostnameFails
  4. OpenClosesTransportAndFreesSslStateWhenHandshakeFailsHard — extends existing OpenFailsImmediatelyOnHardSslError
  5. OpenClosesTransportAndFreesSslStateWhenHandshakeBudgetExhausts — extends existing OpenFailsWhenHandshakeNeverCompletes
  6. SecondOpenAfterFailedFirstOpenSucceeds — recovery proof: first handshake fails, second succeeds; asserts Transport.OpenCount == 2, Transport.CloseCount == 1, second Open returns true.
• All existing tests still pass; in particular, CloseAfterInternalCloseFromSendFailureDoesNotDoubleFree still pins the idempotent-Close invariant.

Verification (developer-side, not a CI gate)

Run the untrusted-CA integration test (HandshakeFailsWhenServerCertSignedByUntrustedCa) in a loop with ulimit -n 32 and confirm it does not hit EMFILE. Record the result in the PR description.

Out of scope

• OpenSSL parity. TlsStream_Open has the same shape; ships as S26.03 after this story merges, using the test pattern proven here.
• Defensive close-before-open in PosixTcpStream_Open. The contract today is "every Stream_Open is preceded by Stream_Close"; StreamSender honours it via Disconnect-before-reconnect, and the unwind added by this story restores that contract for the mbedTLS failure path. Making PosixTcpStream defensive would paper over future contract violations rather than surface them.
• SolidSyslogWinsockTcpStream and SolidSyslogFreeRtosTcpStream hardening. Same clobber-on-reopen shape, same argument: not reachable from a valid caller once mbedTLS unwinds correctly.
• StreamSender reconnect flow changes.