Remote Process-Crash DoS via Reachable `assert!` in `poll_app_data`

[X1AOxiang]

Summary

A reachable assert! in dimpl's application-data delivery path can be triggered by any authenticated peer, crashing the host process and every other DTLS connection it carries. The inbound record-length is bounded only by the 16-bit DTLS record-length field, not by config.mtu() or any
inbound maximum. When the integrator calls poll_output with a buffer smaller than a received ApplicationData fragment (the documented example uses a 2048-byte buffer), the assertion fires and the process aborts.

Affected Versions

• dimpl 0.6.2 (revision dd5a20f, 0.6.2-7-gdd5a20f)
• Both DTLS 1.2 and DTLS 1.3 engines.

Severity

HIGH — remote denial of service, process crash. Trivially triggerable by any authenticated peer (a client whose certificate or PSK the server
accepts — routine in WebRTC). One UDP datagram kills the process and all co-located connections.

CWE

• CWE-617 (Reachable Assertion)
• CWE-248 (Uncaught Exception / Abort)
• CWE-400 (Uncontrolled Resource Consumption)

Root Cause

Engine::poll_app_data uses an assert! to check that the integrator's output buffer is large enough to hold a decrypted ApplicationData fragment. There is no inbound record-length cap anywhere on the receive path — config.mtu() is outbound-only — so a peer can deliver a fragment up to ~65535 bytes, far exceeding the buffer sizes integrators typically allocate (the crate's own documented example uses vec![0u8; 2048]).

Vulnerable code

DTLS 1.3 — src/dtls13/engine.rs (line 580):

fn poll_app_data<'a>(&mut self, buf: &'a mut [u8]) -> Result<&'a [u8], &'a mut [u8]> {
    // ...
    let len = fragment.len();
    assert!(
        len <= buf.len(),
        "Output buffer too small for application data {} > {}",
        len,
        buf.len()
    );
    buf[..len].copy_from_slice(fragment);
    // ...
}

DTLS 1.2 — src/dtls12/engine.rs (line 459, mirror with the same assert!).

Attack Chain

1. The attacker completes one normal DTLS 1.2 (or 1.3) handshake with the victim server. This is the only precondition — an authenticated peer, trivially attainable whenever the server accepts self-signed certificates or a known PSK (common in WebRTC).
2. The attacker sends a single UDP datagram containing one encrypted ApplicationData record whose 16-bit length field declares a large value (e.g. ~40000 bytes), carrying legitimately AEAD-encrypted application data of that size. The whole datagram fits within the UDP/IPv4 datagram limit (~64 KB).
3. handle_packet → Incoming::parse_packet → Records::parse: record_end = header_len + length; there is no inbound MTU/max-length gate. Record::parse copies the whole record into a growable Buf.
4. Decryption succeeds (the attacker holds the key) → a large plaintext fragment lands in queue_rx.
5. The integrator's event loop calls poll_output(&mut out_buf) with a buffer sized to the documented example (2048 bytes) or to config.mtu()
   (1150 bytes).
6. fragment.len() > buf.len() → assert!(len <= buf.len()) fails → panic → process abort. Every other DTLS connection the process was carrying dies with it.

Proof of Concept

The PoC is supplied as a Git patch (poc.patch) that adds one new test file tests/dtls12/poc.rs and registers it in tests/dtls12/main.rs.
The test poc_v1_oversized_appdata_panics uses only the public Sans-IO API.

Applying the patch

From the repository root (dimpl/, at revision dd5a20f):

git apply poc.patch

Running the PoC

cargo test --features rcgen --test dtls12 poc::poc_v1_oversized_appdata_panics -- --nocapture

What the test does

1. Complete a DTLS 1.2 handshake between two Dtls endpoints (client configured with a 9000-byte MTU so a single ApplicationData record
   covers the whole payload).

2. The client sends 4000 bytes of application data.

3. The server accepts and decrypts the oversized record without error.

4. The server's poll_output is called with a 2048-byte buffer (the size used in the crate's documented example).

5. poll_output panics with:

   Output buffer too small for application data 4000 > 2048
   

The test wraps the call in std::panic::catch_unwind and asserts that a panic occurs, confirming the process-crash DoS.

Verified output

test poc::poc_v1_oversized_appdata_panics ... ok

The panic message Output buffer too small for application data 4000 > 2048 is printed to stderr, confirming the assert! at src/dtls12/engine.rs:459 fires.

Impact

• Remote DoS: any authenticated peer can crash the server process with a single UDP datagram.
• Blast radius: the crash kills the entire process, including every other DTLS connection it hosts. For a WebRTC SFU or gateway this means
  one malicious client can disconnect every other participant.

Suggested Fix

1. Replace the assert!(len <= buf.len()) in poll_app_data (and the symmetric check in poll_packet_tx) with a returned Error (e.g. a new Error::OutputBufferTooSmall) so the caller can retry with a larger buffer instead of crashing.
2. Add an inbound record-length cap in Records::parse / Record::parse: reject any record whose declared length exceeds a constant maximum
   (e.g. 16384, matching RFC 8446/9147 record-size conventions).
3. Expose a Dtls::buffer_size_hint() (or document the required minimum poll_output buffer size) so integrators can size their buffers correctly without guessing.
4. Update the poll_output documentation and the documented example to use a correctly-sized buffer.

poc.patch

[algesten]

Hi! Thanks for submitting this.

HIGH — remote denial of service, process crash. Trivially triggerable by any authenticated peer (a client whose certificate or PSK the server
accepts — routine in WebRTC). One UDP datagram kills the process and all co-located connections.

I disagree with calling this HIGH, for two reasons.

1. Application data is not fragmented in DTLS, and while IP fragmentation is a thing, it's de-facto not really a thing. The chances of you successfully delivering a an oversized application data packet on the regular internet are slim.

2. The buffer you poll the data into is use controlled. A conservative choice would be to always use like 16kb buffers even if you only expect 2k due to MTU.

Having said that, we will make the code better.