memory exhaustion in ReadTIFFImage

[bestshow]

Version: ImageMagick 7.0.7-1 Q16 x86_64
#./convert $FILE /dev/null
When convert tiff file , imagemagick will allocate memory to store the data, here is the critical code:
(tiff.c , in function ReadTIFFImage)

        tile_pixels=(uint32 *) AcquireQuantumMemory(columns,rows*  //1934
          sizeof(*tile_pixels));

The "rows" can be got from tiff file,that is to say it can be controlled by input file.Here is my policy.xml to limit memory usage,but 256MB limit can be bypassed.

...
<policy domain="resource" name="area" value="100MP"/>
<policy domain="resource" name="memory" value="256MiB"/>
...

testcase:https://github.com/bestshow/p0cs/blob/master/memory_exhaustion_in_ReadTIFFImage_1934
Credit:ADLab of Venustech

[asarubbo]

see #269

[mikayla-grace]

Thanks for the problem report. We can reproduce it and will have a patch to fix it in GIT master branch @ https://github.com/ImageMagick/ImageMagick later today. The patch will be available in the beta releases of ImageMagick @ http://www.imagemagick.org/download/beta/ by sometime tomorrow.

[nohmask]

This was assigned CVE-2017-18028.