Heap OOB reads in pep_deserialize and pep_decompress (CWE-125/CWE-122)

[ByamB4]

Summary

pep_deserialize() accepts no input length parameter, so every read from bytes_ref is unbounded. A crafted or truncated .pep file causes heap out-of-bounds reads throughout the parsing. Additionally, pep_decompress() can produce OOB palette accesses from crafted compressed data.

Vulnerability Details

Bug 1: pep_deserialize has no input length — all reads are unbounded (line 883)

static inline pep pep_deserialize( const uint8_t* const in_bytes )
{
    const uint8_t* bytes_ref = in_bytes;
    // ...
    uint8_t packed_flags = *bytes_ref++;   // unchecked
    // ... width/height reads (unchecked)
    // ... palette reads (unchecked)
}

The function takes a pointer with no size. Every *bytes_ref++ (flags byte at line 892, width/height at 903-910, palette entries at 949-995) reads without verifying remaining input. A 2-byte .pep file triggers OOB reads on every subsequent field.

Bug 2: Attacker-controlled memcpy length from untrusted bytes_size (line 1000-1003)

out_pep.bytes = (uint8_t*)PEP_MALLOC(bytes_size);
if (out_pep.bytes) {
    memcpy(out_pep.bytes, bytes_ref, bytes_size);  // bytes_size from input
}

bytes_size is parsed from the file's variable-length encoding (attacker-controlled). If it exceeds the remaining input data, memcpy reads past the allocation into adjacent heap memory. A .pep file claiming bytes_size = 0x7FFFFFFF but containing only a few bytes triggers a massive heap over-read.

Bug 3: Variable-length size parsing — shift overflow and unbounded reads (lines 921-927)

do {
    byte_val = *bytes_ref++;              // no bounds check
    bytes_size |= ((uint32_t)(byte_val & 0x7f)) << shift;
    shift += 7;
} while (byte_val & 0x80);

• Unbounded reads: if all bytes have bit 7 set, the loop reads indefinitely past the input buffer.
• Undefined behavior: after 5 iterations, shift = 35. Left-shifting a uint32_t by 35 is undefined behavior per C standard (shift amount >= type width).

Bug 4: OOB symbol from arithmetic decoder → palette OOB (lines 335-347, 685)

In _pep_get_sym_from_freq:

for (; s <= PEP_FREQ_END; ++s) {    // PEP_FREQ_END = 256
    freq += ctx->freq[s];
    if (freq > target_freq) break;
}
// If loop completes without break: s = 257
result.prob.low = freq - ctx->freq[s];  // freq[257] is OOB (array has 257 elements)
result.symbol = s;                       // symbol = 257

With crafted compressed data where target_freq >= ctx->sum, the loop exits with s = 257. This causes:

1. ctx->freq[257] — OOB read (1 past the 257-element array)
2. In pep_decompress line 685: palette[decode_result.symbol] where symbol = 257 — OOB read past the 256-element palette array

Impact

Any application using pep_load() or pep_deserialize() to load untrusted .pep files is vulnerable. A crafted file can:

• Read arbitrary heap memory (information disclosure)
• Crash the application via segfault
• Potentially achieve code execution via heap corruption (if writes follow OOB reads)

Suggested Fix

1. Add an in_bytes_size parameter to pep_deserialize and check remaining bytes before every read
2. Limit the variable-length size loop to at most 5 iterations (covers 32-bit range) and cap shift < 32
3. In _pep_get_sym_from_freq, return an error or clamp s to PEP_FREQ_END after the loop
4. In pep_decompress, validate that decoded symbols are within palette_size before indexing

Found via manual code audit.