MQTT publish: remaining-length arithmetic wraps on 32-bit size_t -> small allocation + huge memcpy (heap overflow window)

[bupt-Yy-young]

Environment: curl 8.22.0 and current master (lib/mqtt.c mqtt_publish()), 32-bit builds.
64-bit builds are not affected.

What happens

For an mqtt:// POST, mqtt_publish() computes the packet "remaining length" as
payloadlen + 2 + topiclen in size_t. On a 32-bit build, payloadlen can be as large as
SIZE_MAX (0xFFFFFFFF) via CURLOPT_POSTFIELDSIZE_LARGE after curlx_sotouz_fits() -- the value
"fits" because it is not greater than SIZE_MAX. The addition then wraps around 32 bits:

• payloadlen = 0xFFFFFFFF, topiclen = 4
• remaininglength = 0xFFFFFFFF + 2 + 4 = 5 (mod 2^32)

The guard that should reject oversized messages checks the wrapped value, so it passes:

if(remaininglength > (MAX_MQTT_MESSAGE_SIZE - encodelen - 1)) ... // 5 < 0xFFFFFFF -> passes

curl then allocates a tiny buffer (remaininglength + 1 + encodelen, here 7 bytes) and afterwards
does memcpy(&pkt[i], payload, payloadlen) with the unwrapped payloadlen (4 GB) -- a heap
overflow window (and an out-of-bounds read of the caller's buffer) on 32-bit.

Verified by compiling the exact arithmetic for 32-bit (size_t = 4 bytes):
postfieldsize=0xFFFFFFFF -> remaininglength wraps to 5 -> 7-byte allocation -> memcpy of 4294967295 bytes.

Conditions

• 32-bit build (size_t 32-bit).
• mqtt:// URL with POST, CURLOPT_POSTFIELDS non-NULL and CURLOPT_POSTFIELDSIZE_LARGE set to

  = ~0xFFFFFFFD (the curl CLI cannot reach this; only a libcurl application passing such a size is
  affected). It still needs an app-supplied size that is wrong for its actual buffer, so practical
  exploitability is low -- but the arithmetic overflow itself is real and the size guard is defeated.

Why

• lib/mqtt.c, mqtt_publish(): remaininglength = payloadlen + 2 + topiclen (size_t, wraps on
  32-bit) is computed before the MAX_MQTT_MESSAGE_SIZE check, so the check operates on the
  wrapped value.
• Note the receive path is fine: mqtt_decode_len() caps at the protocol maximum 0xFFFFFFF which
  fits 32-bit size_t; this is purely the send/publish path.

Expected vs actual

Expected: an oversized payload is rejected before any wrap (CURLE_TOO_LARGE).
Actual: on 32-bit the wrap defeats the guard and leads to a tiny allocation followed by a huge
memcpy.

Fix direction: validate the size in 64-bit arithmetic (or check payloadlen against
MAX_MQTT_MESSAGE_SIZE before the wrap) before allocating and copying.

[bagder]

I'm a little curious how you have a 4GB buffer in memory on a 32-bit system...