DTLS: cap concurrent half-open handshakes (pre-cookie state-exhaustion) - #2012
Merged
Conversation
A DTLS ClientHello from a new source makes the listener allocate a per-peer SSL + ioa_socket + ts_ur_super_session in dtls_server_input_handler before the source has answered the RFC 6347 cookie challenge. A cookie-less ClientHello only elicits a HelloVerifyRequest, which a source-spoofing attacker or botnet never answers, so one datagram per forged source accumulated durable state with no bound until the allocate timeout - GHSA-5x2p-4vqj-f6m4 (CWE-770 / CWE-400). Measured ~120 kB/source, RSS growing linearly to OOM. Bound the number of concurrent half-open (handshake-incomplete) DTLS sockets across all relay threads with a global atomic counter (turn_dtls_half_open). The cap scales with server size - TURN_DTLS_HALF_OPEN_PER_THREAD (16) times the relay-thread count - so a bigger deployment tolerates proportionally more concurrent handshakes and a small box is not over-committed. A single global counter (rather than a hard per-thread partition) lets a busy relay thread use headroom left idle by others while the process-wide ceiling still bounds total memory. The listener reserves a slot before allocating any per-peer state and drops the datagram when the cap is reached; the slot is released when the handshake completes (data path) or the socket is closed (close_ioa_socket, covering handshakes that never finish and are reaped by the allocate timeout). A per-socket dtls_half_open flag makes the release idempotent. This does not touch the DTLS cookie/handshake flow, so legitimate handshakes are unchanged - a completing client releases its slot within a few round trips, so the cap only bites under a flood. Turns an unbounded memory-exhaustion DoS into a fixed ceiling: measured ~7.9 MB total at 10 000 spoofed sources with 4 relay threads (cap 64), vs ~1.2 GB unbounded. Also stops a DTLS handshake datagram that yields no socket (cap reached or handshake error) from falling through to create_ioa_socket_from_fd and becoming a plain-UDP session. Builds on the DTLS_OTHER drop in the previous commit. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
eakraly
marked this pull request as ready for review
July 26, 2026 22:18
eakraly
added a commit
that referenced
this pull request
Aug 3, 2026
* DTLS: verify the cookie statelessly (DTLSv1_listen) before allocating state The DTLS listener created a per-peer SSL + ioa_socket + ts_ur_super_session for every ClientHello from a new source, before the RFC 6347 section 4.2.1 cookie was verified. A source-spoofing flood of cookieless ClientHellos thus accumulated per-source state - GHSA-5x2p-4vqj-f6m4. The half-open cap (#2012) bounded the total but did not remove it: a flood could still saturate the cap and deny handshakes to legitimate DTLS clients. Run the cookie exchange through OpenSSL's stateless DTLSv1_listen() instead. The ClientHello already read off the shared listener fd is wrapped in a memory BIO (rbio); a datagram BIO addressed to the peer (wbio) carries the HelloVerifyRequest. A ClientHello without a valid cookie only elicits the HelloVerifyRequest and the throwaway SSL is freed immediately - zero retained state. Only a cookie-verified ClientHello, which proves the source can receive at its claimed address, is promoted to a real socket + session; the half-open cap now applies only to these return-routable peers. One SSL_do_handshake() emits the ServerHello flight so the handshake does not wait for a retransmit; the rest completes over the child socket's existing read path. This is why the earlier "gate on a cookie-verified flag with a fresh SSL" attempt could not work: a fresh SSL cannot resume the manual SSL_read cookie exchange (the client's second ClientHello carries message_seq=1). DTLSv1_listen is stateless about the cookie (an HMAC of the peer address), so a fresh SSL verifies the second ClientHello correctly. Feasible now that master is OpenSSL-3.0+-only (#2011): DTLSv1_listen and BIO_ADDR are uniform 3.x APIs. The now-unused dtls_accept_client_connection() is removed. Validated (OpenSSL 3.x/macOS): DTLS relay handshake completes end-to-end (run_tests.sh -S, run_tests_dtls_default.sh); a 10 000-source cookieless flood grows RSS 0.5 MB (allocator noise) vs 6.9 MB on the cap-only build with the identical flood; unit tests 18/18; run_tests.sh TCP/TLS/UDP/DTLS all pass. Not yet run: the Linux/Docker validation matrix. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * Update dtls_listener.c --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A DTLS ClientHello from a new source makes the listener allocate a per-peer SSL + ioa_socket + ts_ur_super_session in dtls_server_input_handler before the source has answered the RFC 6347 cookie challenge. A cookie-less ClientHello only elicits a HelloVerifyRequest, which a source-spoofing attacker or botnet never answers, so one datagram per forged source accumulated durable state with no bound until the allocate timeout
Bound the number of concurrent half-open (handshake-incomplete) DTLS sockets across all relay threads with a global atomic counter (turn_dtls_half_open). The cap scales with server size - TURN_DTLS_HALF_OPEN_PER_THREAD (16) times the relay-thread count - so a bigger deployment tolerates proportionally more concurrent handshakes and a small box is not over-committed. A single global counter (rather than a hard per-thread partition) lets a busy relay thread use headroom left idle by others while the process-wide ceiling still bounds total memory.
The listener reserves a slot before allocating any per-peer state and drops the datagram when the cap is reached; the slot is released when the handshake completes (data path) or the socket is closed (close_ioa_socket, covering handshakes that never finish and are reaped by the allocate timeout). A per-socket dtls_half_open flag makes the release idempotent.
This does not touch the DTLS cookie/handshake flow, so legitimate handshakes are unchanged - a completing client releases its slot within a few round trips, so the cap only bites under a flood.
Also stops a DTLS handshake datagram that yields no socket (cap reached or handshake error) from falling through to create_ioa_socket_from_fd and becoming a plain-UDP session.
Builds on the DTLS_OTHER drop in the previous commit.