TLS handshake failure sends close_notify instead of fatal alert

[He-Pin]

Description

When a TLS handshake fails, both the classic remoting and stream TLS implementations send a close_notify alert (graceful shutdown) instead of a fatal TLS alert as required by RFC 5246 §7.2.1 and RFC 8446 §6.1.

Affected code

Classic remoting — remote/src/main/scala/org/apache/pekko/remote/transport/netty/NettySSLSupport.scala (lines 83-92):

handler.handshakeFuture().addListener((future: Future[Channel]) => {
  if (!future.isSuccess) {
    val channel = handler.closeOutbound().channel()  // sends close_notify
    channel.close()
  }
})

Stream TLS GraphStage — stream/src/main/scala/org/apache/pekko/stream/impl/io/TlsGraphStage.scala (lines 332-340, 599-610):

case ex: SSLException =>
  failTls(ex, closeTransport = false)  // triggers writeFailureAlert()
  // writeFailureAlert() calls engine.closeOutbound() → close_notify

Legacy stream TLSActor — stream/src/main/scala/org/apache/pekko/stream/impl/io/TLSActor.scala (lines 312-318):

case ex: SSLException =>
  fail(ex, closeTransport = false)
  engine.closeInbound()
  completeOrFlush()  // no explicit fatal alert

Analysis

• close_notify is defined for orderly shutdown of established connections, not for handshake failures
• Per TLS spec, failed handshakes should produce a fatal alert (handshake_failure, bad_certificate, etc.)
• Sending close_notify instead of a fatal alert can cause the peer to wait indefinitely for handshake completion
• The non-TLS TCP code correctly uses Abort (TCP reset) for failure scenarios, but the TLS code does not follow this pattern

Suggested fix

On handshake failure:

1. Skip closeOutbound() / writeFailureAlert()
2. Perform an abrupt TCP close (e.g., channel.close() directly without TLS close_notify)
3. This aligns with the non-TLS path in TcpStages.scala which uses Abort for failures

[pjfanning]

how does this differ from #3237