memory exhaustion in ReadOneLayer

[whiteHat001]

root@ubuntu:/home/hjy/Desktop# magick --version
Version: ImageMagick 7.0.6-6 Q16 i686 2017-08-05 http://www.imagemagick.org
Copyright: © 1999-2017 ImageMagick Studio LLC
License: http://www.imagemagick.org/script/license.php
Features: Cipher DPC HDRI OpenMP
Delegates (built-in): jng jpeg png xml zlib
root@ubuntu:/home/hjy/Desktop# magick convert oom-ReadOneLayer /dev/null
==9122==ERROR: AddressSanitizer failed to allocate 0x156c9000 (359436288) bytes of LargeMmapAllocator: 12
==9122==Process memory map follows:
0x08048000-0x094d3000 /usr/local/bin/magick
0x094d3000-0x094d4000 /usr/local/bin/magick
0x094d4000-0x09555000 /usr/local/bin/magick
0x09555000-0x09558000
0x0a936000-0x1ffff000 [heap]
0x1ffff000-0x24000000
0x24000000-0x28000000
0x28000000-0x40000000
0x4b1a5000-0x8b700000
0x8b72f000-0xa0df8000
0xa0df8000-0xa0df9000
0xa0df9000-0xa15f9000
0xa177c000-0xa177d000
0xa177d000-0xa1f7d000
0xa2100000-0xa2200000
0xa227c000-0xa227d000
0xa227d000-0xa2d00000 [stack:9174]
0xb2f00000-0xb3000000
0xb3100000-0xb3200000
0xb3300000-0xb3400000
0xb3500000-0xb3600000
0xb3700000-0xb3800000
0xb3900000-0xb3a00000
0xb3b00000-0xb3c00000
0xb3d00000-0xb3e00000
0xb3f00000-0xb4100000 /usr/lib/locale/locale-archive
0xb4100000-0xb4200000
0xb4300000-0xb4400000
0xb4500000-0xb4600000
0xb4700000-0xb4800000
0xb4900000-0xb4a00000
0xb4b00000-0xb4c00000
0xb4d00000-0xb4e00000
0xb4f00000-0xb5000000
0xb5100000-0xb5200000
0xb5300000-0xb5400000
0xb5500000-0xb5600000
0xb5700000-0xb5800000
0xb5900000-0xb5a00000
0xb5ad0000-0xb5c00000
0xb5c09000-0xb5c6e000
0xb5c6e000-0xb5c6f000 /usr/lib/locale/locale-archive
0xb5c6f000-0xb6e22000
0xb6e22000-0xb6e46000 /lib/i386-linux-gnu/liblzma.so.5.0.0
0xb6e46000-0xb6e47000 /lib/i386-linux-gnu/liblzma.so.5.0.0
0xb6e47000-0xb6e48000 /lib/i386-linux-gnu/liblzma.so.5.0.0
0xb6e48000-0xb6e4b000 /lib/i386-linux-gnu/libdl-2.19.so
0xb6e4b000-0xb6e4c000 /lib/i386-linux-gnu/libdl-2.19.so
0xb6e4c000-0xb6e4d000 /lib/i386-linux-gnu/libdl-2.19.so
0xb6e4d000-0xb6ff6000 /lib/i386-linux-gnu/libc-2.19.so
0xb6ff6000-0xb6ff8000 /lib/i386-linux-gnu/libc-2.19.so
0xb6ff8000-0xb6ff9000 /lib/i386-linux-gnu/libc-2.19.so
0xb6ff9000-0xb6ffc000
0xb6ffc000-0xb7014000 /lib/i386-linux-gnu/libpthread-2.19.so
0xb7014000-0xb7015000 /lib/i386-linux-gnu/libpthread-2.19.so
0xb7015000-0xb7016000 /lib/i386-linux-gnu/libpthread-2.19.so
0xb7016000-0xb7018000
0xb7018000-0xb7033000 /lib/i386-linux-gnu/libgcc_s.so.1
0xb7033000-0xb7034000 /lib/i386-linux-gnu/libgcc_s.so.1
0xb7034000-0xb7035000 /lib/i386-linux-gnu/libgcc_s.so.1
0xb7035000-0xb7036000
0xb7036000-0xb707a000 /lib/i386-linux-gnu/libm-2.19.so
0xb707a000-0xb707b000 /lib/i386-linux-gnu/libm-2.19.so
0xb707b000-0xb707c000 /lib/i386-linux-gnu/libm-2.19.so
0xb707c000-0xb70aa000 /usr/lib/i386-linux-gnu/libgomp.so.1.0.0
0xb70aa000-0xb70ab000 /usr/lib/i386-linux-gnu/libgomp.so.1.0.0
0xb70ab000-0xb70ac000 /usr/lib/i386-linux-gnu/libgomp.so.1.0.0
0xb70ac000-0xb70c4000 /lib/i386-linux-gnu/libz.so.1.2.8
0xb70c4000-0xb70c5000 /lib/i386-linux-gnu/libz.so.1.2.8
0xb70c5000-0xb70c6000 /lib/i386-linux-gnu/libz.so.1.2.8
0xb70c6000-0xb721b000 /usr/lib/i386-linux-gnu/libxml2.so.2.9.1
0xb721b000-0xb721c000 /usr/lib/i386-linux-gnu/libxml2.so.2.9.1
0xb721c000-0xb7220000 /usr/lib/i386-linux-gnu/libxml2.so.2.9.1
0xb7220000-0xb7221000 /usr/lib/i386-linux-gnu/libxml2.so.2.9.1
0xb7221000-0xb7222000
0xb7222000-0xb7248000 /lib/i386-linux-gnu/libpng12.so.0.50.0
0xb7248000-0xb7249000 /lib/i386-linux-gnu/libpng12.so.0.50.0
0xb7249000-0xb724a000 /lib/i386-linux-gnu/libpng12.so.0.50.0
0xb724a000-0xb724b000
0xb724b000-0xb7294000 /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
0xb7294000-0xb7295000 /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
0xb7295000-0xb7296000 /usr/lib/i386-linux-gnu/libjpeg.so.8.0.2
0xb7296000-0xb72a6000
0xb72a6000-0xb7345000 /usr/lib/i386-linux-gnu/libasan.so.1.0.0
0xb7345000-0xb7347000 /usr/lib/i386-linux-gnu/libasan.so.1.0.0
0xb7347000-0xb7348000 /usr/lib/i386-linux-gnu/libasan.so.1.0.0
0xb7348000-0xb77a1000
0xb77a1000-0xb77b7000
0xb77b7000-0xb77b8000 [vdso]
0xb77b8000-0xb77d8000 /lib/i386-linux-gnu/ld-2.19.so
0xb77d8000-0xb77d9000 /lib/i386-linux-gnu/ld-2.19.so
0xb77d9000-0xb77da000 /lib/i386-linux-gnu/ld-2.19.so
0xbf957000-0xbf978000 [stack]
==9122==End of process memory map.
==9122==AddressSanitizer CHECK failed: ../../../../src/libsanitizer/sanitizer_common/sanitizer_posix.cc:66 "(("unable to mmap" && 0)) != (0)" (0x0, 0x0)
#0 0xb72fa4c1 (/usr/lib/i386-linux-gnu/libasan.so.1+0x544c1)
#1 0xb72fe6a9 in __sanitizer::CheckFailed(char const*, int, char const*, unsigned long long, unsigned long long) (/usr/lib/i386-linux-gnu/libasan.so.1+0x586a9)
#2 0xb7303e22 (/usr/lib/i386-linux-gnu/libasan.so.1+0x5de22)
#3 0xb72be99b (/usr/lib/i386-linux-gnu/libasan.so.1+0x1899b)
#4 0xb72bf5e9 (/usr/lib/i386-linux-gnu/libasan.so.1+0x195e9)
#5 0xb72f4d71 in __interceptor_posix_memalign (/usr/lib/i386-linux-gnu/libasan.so.1+0x4ed71)
#6 0x80e587d in AcquireAlignedMemory MagickCore/memory.c:262
#7 0x87fa923 in OpenPixelCache MagickCore/cache.c:3575
#8 0x80607bc in GetImagePixelCache MagickCore/cache.c:1654
#9 0x881216d in SyncImagePixelCache MagickCore/cache.c:5269
#10 0x809a538 in SetImageStorageClass MagickCore/image.c:2513
#11 0x809a538 in SetImageBackgroundColor MagickCore/image.c:2322
#12 0x86665f0 in ReadOneLayer coders/xcf.c:917
#13 0x86665f0 in ReadXCFImage coders/xcf.c:1331
#14 0x88980c8 in ReadImage MagickCore/constitute.c:497
#15 0x889bb49 in ReadImages MagickCore/constitute.c:866
#16 0x8ea0ba0 in ConvertImageCommand MagickWand/convert.c:641
#17 0x8fa97d1 in MagickCommandGenesis MagickWand/mogrify.c:183
#18 0x8074e7a in MagickMain utilities/magick.c:149
#19 0x805572a in main utilities/magick.c:180
#20 0xb6e66a82 in __libc_start_main (/lib/i386-linux-gnu/libc.so.6+0x19a82)
#21 0x80744ba (/usr/local/bin/magick+0x80744ba)
POC https://github.com/whiteHat001/FUZZ_POC/blob/master/oom-ReadOneLayer

[mikayla-grace]

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ http://www.imagemagick.org/download/beta/ by sometime tomorrow.

[fgeek]

Please use CVE-2017-12691 for this issue.