memory exhaustion in ReadPCXImage

[bestshow]

Version: ImageMagick 7.0.6-1 Q16 x86_64
#./magick identify $FILE
When identify PCX file , imagemagick will allocate memory to store the data.
Here is the critical code:

pixel_info=AcquireVirtualMemory(pcx_packets,2*sizeof(*pixels)); //406

pcx_packets can be controlled as follow:

  pcx_packets=(size_t) image->rows*pcx_info.bytes_per_line; //397
  ...
  pcx_packets=(size_t) pcx_packets*pcx_info.planes; //400

pcx_info.planes can be read from input file

pcx_info.planes=(unsigned char) ReadBlobByte(image); //358

Here is my policy.xml to limit memory usage,but 256MB limit can be bypassed.

...
<policy domain="resource" name="area" value="100MP"/>
<policy domain="resource" name="memory" value="256MiB"/>
...

testcase:https://github.com/bestshow/p0cs/blob/master/memory_exhaustion_in_ReadPCXImage
Creadit : ADLab of Venustech

[mikayla-grace]

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ http://www.imagemagick.org/download/beta/ by sometime tomorrow.

[nohmask]

This was assigned CVE-2017-12432.