Prevent signed integer overflow in compound dictionary total_size

[mohammadmseet-hue]

Summary

AttachCompoundDictionary() in c/dec/decode.c accumulates dictionary chunk sizes into total_size (type int) without overflow checking. With up to 15 chunks allowed and no limit on individual chunk sizes, cumulative sizes exceeding INT_MAX cause signed integer overflow — undefined behavior in C.

The Bug

Line 1545:

addon->total_size += (int)size;

total_size is int (32-bit signed). size is size_t. With 15 chunks of 256MB each:

Chunk  7: total = 1,879,048,192 (fits in int)
Chunk  8: total = 2,147,483,648 (exceeds INT_MAX → signed overflow → UB)

UBSAN Confirmation

c/dec/decode.c:1545:21: runtime error: signed integer overflow:
1879048192 + 268435456 cannot be represented in type 'int'

Downstream Impact

The overflowed total_size propagates to:

• chunk_offsets[] (line 1546) — stores incorrect accumulated offsets
• EnsureCompoundDictionaryInitialized() (line 1557) — (addon->total_size - 1) >> block_bits with negative total_size causes incorrect block mapping
• CopyFromCompoundDictionary() (line 1598) — computes rem_chunk_length from corrupted chunk_offsets, potentially yielding incorrect memcpy lengths

Note: the encoder side (encode.c) correctly uses size_t for the equivalent computation, confirming this is an oversight in the decoder.

The Fix

Two overflow checks added before the addition:

1. Reject individual chunks larger than 0x7FFFFFFF (INT_MAX)
2. Reject chunks that would cause total_size to exceed 0x7FFFFFFF

if (size > (size_t)0x7FFFFFFF) return BROTLI_FALSE;
if ((int)size > 0x7FFFFFFF - addon->total_size) return BROTLI_FALSE;

Reproduction

BrotliDecoderState* state = BrotliDecoderCreateInstance(NULL, NULL, NULL);
void* buf = mmap(NULL, 256*1024*1024, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0);
for (int i = 0; i < 15; i++) {
    BrotliDecoderAttachDictionary(state, BROTLI_SHARED_DICTIONARY_RAW, 256*1024*1024, buf);
    // Chunk 8 triggers: total_size overflow from 1,879,048,192 + 268,435,456
}

Build with: gcc -fsanitize=undefined to see the UBSAN error.

[eustas]

Hi. Thanks for reporting. Would you mind if I make an alternative fix? (of course, this PR will be mentioned)

[mohammadmseet-hue]

Yes of course, implement the fix the way you like, happy to help and would appreciate the mention

[eustas]

Has been landed (merged) via #1450

[mohammadmseet-hue]

Dear @eustas , you didn't mention my name correctly in 1450, you mentioned someone else's name and related it to my PR. Kindly your support as it matters.

[eustas]

Very sorry. Will fix that ASAP.