integer overflow in GC_MALLOC_ATOMIC #135
When call GC_MALLOC_ATOMIC(0xFFFFFFFFFFFFF2ABull), the expected behavior should be out of memory obviously and return NULL or abort. However, libgc returns a pointer. The caller thought the allocation succeeded and started to write data into heap via the said pointer and thus heap corruption.
The reason is integer overflow in macro
(sz) + HBLKSIZE-1 overflows and become a small positive number. After the overflow, libgc allocates a small block of memory and return the pointer.
referenced this issue
Aug 21, 2016
This issue was assigned CVE-2016-9427 ; see http://www.openwall.com/lists/oss-security/2016/11/18/3 for reference.