Properly handle SSL handshake updates

[cshannon]

Handle different handshake state changes in NIOSSL transports

Edit:

Here is some more info on the fixes now that CVE-2026-39304 is published:

The quick version is this was a bug related to if a client requests a key update or handshake renegotiation depending on TLS version. The code was not handling the NEED_WRAP, NEED_TASK or FINISHED status correctly during renegotiation which can cause issues including OOM or stuck connections depending on the TLS version used. The code assumed after the initial handshake was done that was it. The updates here now correctly detect those states after the initial handshake and process.

Summary of issues/changes:

1. If using TLSv1.3, When a client requests a handshake update on the ssl socket, it will trigger a key update. The handshake status on the server will become NEED_WRAP and the server needs to respond by calling wrap(), even if there is no application data to send because that triggers the background handshake data to be sent to the client. Our NIO SSL code previously didn't check for this, so wrap() was only called if there was application data to publish back (ie messages to send or heart beats etc). If wrap() is never called (no app data, no heart beating) this can lead to updates building up in memory if a lot of updates are sent by a client.
2. If using TLSv1.2, there is no memory leak because KeyUpdate is not supported. However, the transport can still get stuck. When triggering updates for TLSv1.2 a full renegotiation is triggered and the engine will go into the NEED_TASK status, which is also never handled. So on a TLSv1.2 connection there is no leak but the transport/connection is just stuck and can't do anything. The update now correctly processes NEED_TASK
3. The code originally would only process application data received in the buffer if the handshake state wasn't NEED_UNWRAP, but handshake updates can happen concurrently with application data. This means it's possible for the broker to receive data that needs to be processed while the handshake status state is NEED_UNWRAP. Before this update, it would skip processing any data in the buffer if the state was NEED_UNWRAP which was incorrect.
4. If the state becomes FINISHED, we need to advance the state to NOT_HANDSHAKING and trigger one more read to see if there is more data to process.