Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

API: Prevent large images from repeatedly crashing PHP on resize #2859

Merged
merged 1 commit into from

4 participants

@kinglozzer
Collaborator

See #2753 for more discussion.

This is a (late) follow up to the discussion here about GD crashing when attempting to open large or high DPI images.

The Google Group discussion involved ideas around “image unavailable” icons in the CMS, this setup (by storing which manipulation failed) will hopefully pave the way for that functionality to be added later, while addressing the immediate issue of completely crashing entire sections of websites.

Implements the same method for checking available memory as #2569, though it’s different to the method Ingo suggested. I’m not really sure which is more robust, the method in the php.net comments doesn’t take into account that bits and channels aren’t always present in the image info.

Marked as an API change as I’ve added a few methods to the Image_Backend interface.

Doesn’t cause any issues with the CMS, the images just don’t appear for now:

screen shot 2014-01-02 at 16 15 27
screen shot 2014-01-02 at 16 17 23

All comments welcome.

@Zauberfisch

+1 to get this done as soon as possible

great work @kinglozzer

@kinglozzer
Collaborator

I think it’s probably important that we document this change if it’s accepted, it may be quite confusing to developers otherwise if images disappear (until we set up CMS messages or something).

Would reference/image.md be the best place for this, or would it better to create a topics/image.md? Documentation on this seems more like a ‘topic’ than ‘reference’ to me, but then we’d have two separate documents for the same subject.

@kinglozzer
Collaborator

*bump*

I don’t wanna spend any more time on this until I get a nod of approval from a core committer. Any other comments/criticisms are also welcome of course! :wink:

@simonwelsh simonwelsh added this to the 3.2 alpha 1 milestone
@simonwelsh simonwelsh referenced this pull request in silverstripe/silverstripe-cms
Closed

[2009-02-14] ImageField allowed memory size error #715

@Zauberfisch

bump

can we please get this merged asap?
Especially with those clients we deploy to shared hosting, and the ever increasing image resolution, this is becoming more and more an issue.
I have clients uploading images with a width of 6k, and they can't even delete them anymore because after the error occurred, you never get to a point in the cms where you could delete the image again.

@chillu chillu added the critical label
@kinglozzer
Collaborator

bumpity bump :)

Could this (or any other proposed fix for this issue) make it into 3.2? Not sure how far off we are.

filesystem/GD.php
@@ -65,6 +81,18 @@ public function __construct($filename = null) {
$this->quality = $this->config()->default_quality;
$this->interlace = $this->config()->image_interlace;
}
+
+ /**
+ * If __destruct() is called, any attempted image manipulation must have succeeded,
+ * so it can be removed from the cache

This isn't always true. HHVM at least still calls destructors after a fatal error.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
filesystem/GD.php
@@ -482,6 +560,18 @@ public function writeTo($filename) {
if(file_exists($filename)) @chmod($filename,0664);
}
}
+
+ public function onBeforeDelete($frontend) {
+ $file = Director::baseFolder() . "/" . $frontend->Filename;
+
+ if (file_exists($file)) {
+ $key = md5(implode('_', array($file, filemtime($file))));
+
+ if (unserialize($this->cache->load($key))) {

You can just call remove without the check.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@simonwelsh simonwelsh commented on the diff
tests/filesystem/GDTest.php
((10 lines not shown))
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+ $gd = new GDBackend_ImageUnavailable($fullPath);
+
+ /* Ensure no image resource is created if the image is unavailable */
+ $this->assertNull($gd->getImageResource());
+ }
+
+ /**
+ * Tests the integrity of the manipulation cache when an error occurs
+ * @return void
+ */
+ public function testCacheIntegrity() {
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+
+ try {
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth', 123));

Should have a $this->failure() to make sure an exception's thrown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@simonwelsh simonwelsh commented on the diff
tests/filesystem/GDTest.php
((31 lines not shown))
+
+ $this->assertArrayHasKey('SetWidth|123', $data);
+ $this->assertTrue($data['SetWidth|123']);
+ }
+ }
+
+ /**
+ * Test that GD::failedResample() returns true for the current image
+ * manipulation only if it previously failed
+ * @return void
+ */
+ public function testFailedResample() {
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+
+ try {
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth-failed', 123));

Likewise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@simonwelsh simonwelsh commented on the diff
tests/model/GDImageTest.php
((9 lines not shown))
+ }
+
+ /**
+ * Test that the cache of manipulation failures is cleared when deleting
+ * the image object
+ * @return void
+ */
+ public function testCacheCleaningOnDelete() {
+ $image = $this->objFromFixture('Image', 'imageWithTitle');
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+ $fullPath = $image->getFullPath();
+ $key = md5(implode('_', array($fullPath, filemtime($fullPath))));
+
+ try {
+ // Simluate a failed manipulation
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth', 123));

Likewise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
@kinglozzer
Collaborator

Thanks Simon, I’d not considered that there might be inconsistencies with __destruct() like that. I’m trying to think of an alternative, but it’s quite tricky without making assumptions about how GDBackend might be used: ideally this would also cater for people who use GDBackend outside of the context of Image.

I guess you could argue that we can’t really be responsible for this - we just need to try to prevent, for example, parts of the CMS being inaccessible.

So, should we:

  • A. Not worry about how people may use GDBackend and instead only focus on Image and move some of the logic (i.e. the SS_Cache/stored manipulations logic) into there
  • B. Try to find an alternative to __destruct() and try to keep this as GD-specific as possible
  • C. Forget the __destruct() stuff, just add the approximate memory calculation to GDBackend and see if that resolves the majority of people’s issues
@Zauberfisch

sounds like a tricky issue.

I for one actually do use GBBackend without image as well on rare occasions, and I like to cleanly separate things if possible, but I see that there might not be a way, or it might be a lot harder, so I am fine with B.

Not sure if C is a good solution, but I am not that much into the the whole issue that I could speak to that (I don't know how accurate the approximation is, or how likely/unlikely it is that an error still occurs).
But if we can get C into core significantly faster than A or B; I would love to see C merged and A or B developed further.

@simonwelsh

A couple of options:

  • Remove items from the cache when writeToFile is called
  • Remove items from the cache whenever returning from one of the modifier methods

Given that it's not actually that important to remove items from the cache, I'd prefer the first option.

@kinglozzer
Collaborator

I’ve moved the cache cleaning to writeToFile() and updated a couple of other minor points.

@simonwelsh I’ve added $this->fail() to the unit tests. I wasn’t sure of the correct usage of it: it seems that a try/catch block catches the failure, so I’ve added an assertion for the exception message to make sure it’s highlighted. Is that correct?

@simonwelsh

Would be better to throw a more specific exception (so, a subclass) and then catch that so that the PHPUnit exception doesn't get caught.

@kinglozzer
Collaborator

Tests updated :)

@simonwelsh simonwelsh merged commit 1a63fa5 into silverstripe:master
@simonwelsh

Green button pushed! Now my slides are out of date...

@Zauberfisch

awesome!
great work guys!

@kinglozzer
Collaborator

687474703a2f2f77696c2e746f2f5f2f7965732e676966

Thanks @simonwelsh

@kinglozzer kinglozzer deleted the kinglozzer:gd-resize-crash-mast branch
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
This page is out of date. Refresh to see the latest.
View
2  _config.php
@@ -44,6 +44,8 @@
SS_Cache::add_backend('aggregatestore', 'File', array('cache_dir' => $aggregatecachedir));
SS_Cache::pick_backend('aggregatestore', 'aggregate', 1000);
+SS_Cache::set_cache_lifetime('GDBackend_Manipulations', null, 100);
+
// If you don't want to see deprecation errors for the new APIs, change this to 3.2.0-dev.
Deprecation::notification_version('3.2.0');
View
120 filesystem/GD.php
@@ -8,6 +8,7 @@ class GDBackend extends Object implements Image_Backend {
protected $gd, $width, $height;
protected $quality;
protected $interlace;
+ protected $cache, $cacheKey, $manipulation;
/**
* @config
@@ -34,29 +35,42 @@ public static function set_default_quality($quality) {
}
}
- public function __construct($filename = null) {
+ public function __construct($filename = null, $args = array()) {
// If we're working with image resampling, things could take a while. Bump up the time-limit
increase_time_limit_to(300);
+ $this->cache = SS_Cache::factory('GDBackend_Manipulations');
+
if($filename) {
- // We use getimagesize instead of extension checking, because sometimes extensions are wrong.
- list($width, $height, $type, $attr) = getimagesize($filename);
- switch($type) {
- case 1:
- if(function_exists('imagecreatefromgif'))
- $this->setImageResource(imagecreatefromgif($filename));
- break;
- case 2:
- if(function_exists('imagecreatefromjpeg'))
- $this->setImageResource(imagecreatefromjpeg($filename));
- break;
- case 3:
- if(function_exists('imagecreatefrompng')) {
- $img = imagecreatefrompng($filename);
- imagesavealpha($img, true); // save alphablending setting (important)
- $this->setImageResource($img);
- }
- break;
+ $this->cacheKey = md5(implode('_', array($filename, filemtime($filename))));
+ $this->manipulation = implode('|', $args);
+
+ $cacheData = unserialize($this->cache->load($this->cacheKey));
+ $cacheData = ($cacheData !== false) ? $cacheData : array();
+
+ if ($this->imageAvailable($filename, $this->manipulation)) {
+ $cacheData[$this->manipulation] = true;
+ $this->cache->save(serialize($cacheData), $this->cacheKey);
+
+ // We use getimagesize instead of extension checking, because sometimes extensions are wrong.
+ list($width, $height, $type, $attr) = getimagesize($filename);
+ switch($type) {
+ case 1:
+ if(function_exists('imagecreatefromgif'))
+ $this->setImageResource(imagecreatefromgif($filename));
+ break;
+ case 2:
+ if(function_exists('imagecreatefromjpeg'))
+ $this->setImageResource(imagecreatefromjpeg($filename));
+ break;
+ case 3:
+ if(function_exists('imagecreatefrompng')) {
+ $img = imagecreatefrompng($filename);
+ imagesavealpha($img, true); // save alphablending setting (important)
+ $this->setImageResource($img);
+ }
+ break;
+ }
}
}
@@ -87,6 +101,56 @@ public function getGD() {
}
/**
+ * @param string $filename
+ * @return boolean
+ */
+ public function imageAvailable($filename, $manipulation) {
+ return ($this->checkAvailableMemory($filename) && ! $this->failedResample($filename, $manipulation));
+ }
+
+ /**
+ * Check if we've got enough memory available for resampling this image. This check is rough,
+ * so it will not catch all images that are too large - it also won't work accurately on large,
+ * animated GIFs as bits per pixel can't be calculated for an animated GIF with a global color
+ * table.
+ *
+ * @param string $filename
+ * @return boolean
+ */
+ public function checkAvailableMemory($filename) {
+ $limit = translate_memstring(ini_get('memory_limit'));
+
+ if ($limit < 0) return true; // memory_limit == -1
+
+ $imageInfo = getimagesize($filename);
+
+ // bits per channel (rounded up, default to 1)
+ $bits = isset($imageInfo['bits']) ? ($imageInfo['bits'] + 7) / 8 : 1;
+
+ // channels (default 4 rgba)
+ $channels = isset($imageInfo['channels']) ? $imageInfo['channels'] : 4;
+
+ $bytesPerPixel = $bits * $channels;
+
+ // width * height * bytes per pixel
+ $memoryRequired = $imageInfo[0] * $imageInfo[1] * $bytesPerPixel;
+
+ return $memoryRequired + memory_get_usage() < $limit;
+ }
+
+ /**
+ * Check if this image has previously crashed GD when attempting to open it - if it's opened
+ * successfully, the manipulation's cache key is removed.
+ *
+ * @param string $filename
+ * @return boolean
+ */
+ public function failedResample($filename, $manipulation) {
+ $cacheData = unserialize($this->cache->load($this->cacheKey));
+ return ($cacheData && array_key_exists($manipulation, $cacheData));
+ }
+
+ /**
* Set the image quality, used when saving JPEGs.
*/
public function setQuality($quality) {
@@ -480,6 +544,24 @@ public function writeTo($filename) {
imagepng($this->gd, $filename); break;
}
if(file_exists($filename)) @chmod($filename,0664);
+
+ // Remove image manipulation from cache now that it's complete
+ $cacheData = unserialize($this->cache->load($this->cacheKey));
+ if(isset($cacheData[$this->manipulation])) unset($cacheData[$this->manipulation]);
+ $this->cache->save(serialize($cacheData), $this->cacheKey);
+ }
+ }
+
+ /**
+ * @param Image $frontend
+ * @return void
+ */
+ public function onBeforeDelete($frontend) {
+ $file = Director::baseFolder() . "/" . $frontend->Filename;
+
+ if (file_exists($file)) {
+ $key = md5(implode('_', array($file, filemtime($file))));
+ $this->cache->remove($key);
}
}
View
18 filesystem/ImagickBackend.php
@@ -97,6 +97,16 @@ public function hasImageResource() {
}
/**
+ * @todo Implement memory checking for Imagick? See {@link GD}
+ *
+ * @param string $filename
+ * @return boolean
+ */
+ public function imageAvailable($filename) {
+ return true;
+ }
+
+ /**
* resize
*
* @param int $width
@@ -264,5 +274,13 @@ public function croppedResize($width, $height) {
$new->ThumbnailImage($width,$height,true);
return $new;
}
+
+ /**
+ * @param Image $frontend
+ * @return void
+ */
+ public function onBeforeDelete($frontend) {
+ // Not in use
+ }
}
}
View
8 model/Image.php
@@ -466,7 +466,8 @@ public function generateFormattedImage($format) {
$cacheFile = call_user_func_array(array($this, "cacheFilename"), $args);
$backend = Injector::inst()->createWithArgs(self::$backend, array(
- Director::baseFolder()."/" . $this->Filename
+ Director::baseFolder()."/" . $this->Filename,
+ $args
));
if($backend->hasImageResource()) {
@@ -725,9 +726,12 @@ public function onAfterUpload() {
}
protected function onBeforeDelete() {
- parent::onBeforeDelete();
+ $backend = Injector::inst()->create(self::$backend);
+ $backend->onBeforeDelete($this);
$this->deleteFormattedImages();
+
+ parent::onBeforeDelete();
}
}
View
17 model/Image_Backend.php
@@ -110,4 +110,21 @@ public function paddedResize($width, $height, $backgroundColor = "FFFFFF");
* @return Image_Backend
*/
public function croppedResize($width, $height);
+
+ /**
+ * imageAvailable
+ *
+ * @param string $filename
+ * @return boolean
+ */
+ public function imageAvailable($filename, $manipulation);
+
+ /**
+ * onBeforeDelete
+ *
+ * @param Image $frontend
+ * @return void
+ */
+ public function onBeforeDelete($frontend);
+
}
View
59 tasks/CleanImageManipulationCache.php
@@ -0,0 +1,59 @@
+<?php
+/**
+ * Wipe the cache of failed image manipulations. When {@link GDBackend} attempts to resample an image, it will write
+ * the attempted manipulation to the cache and remove it from the cache if the resample is successful. The objective
+ * of the cache is to prevent fatal errors (for example from exceeded memory limits) from occurring more than once.
+ *
+ * @package framework
+ * @subpackage filesystem
+ */
+class CleanImageManipulationCache extends BuildTask {
+
+ protected $title = 'Clean Image Manipulation Cache';
+
+ protected $description = 'Clean the failed image manipulation cache. Use this to allow SilverStripe to attempt
+ to resample images that have previously failed to resample (for example if memory limits were exceeded).';
+
+ /**
+ * Check that the user has appropriate permissions to execute this task
+ */
+ public function init() {
+ if(!Director::is_cli() && !Director::isDev() && !Permission::check('ADMIN')) {
+ return Security::permissionFailure();
+ }
+
+ parent::init();
+ }
+
+ /**
+ * Clear out the image manipulation cache
+ * @param SS_HTTPRequest $request
+ */
+ public function run($request) {
+ $failedManipulations = 0;
+ $processedImages = 0;
+ $images = DataObject::get('Image');
+
+ if($images && Image::get_backend() == "GDBackend") {
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+
+ foreach($images as $image) {
+ $path = $image->getFullPath();
+
+ if (file_exists($path)) {
+ $key = md5(implode('_', array($path, filemtime($path))));
+
+ if ($manipulations = unserialize($cache->load($key))) {
+ $failedManipulations += count($manipulations);
+ $processedImages++;
+ $cache->remove($key);
+ }
+ }
+ }
+ }
+
+ echo "Cleared $failedManipulations failed manipulations from
+ $processedImages Image objects stored in the Database.";
+ }
+
+}
View
79 tests/filesystem/GDTest.php
@@ -15,6 +15,11 @@ class GDTest extends SapphireTest {
'png32' => 'test_png32.png'
);
+ public function tearDown() {
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+ $cache->clean(Zend_Cache::CLEANING_MODE_ALL);
+ }
+
/**
* Loads all images into an associative array of GD objects.
* Optionally applies an operation to each GD
@@ -135,4 +140,76 @@ function testGreyscale() {
$samplesPNG32 = $this->sampleAreas($images['png32']);
$this->assertGreyscale($samplesPNG32, 8);
}
-}
+
+ /**
+ * Tests that GD doesn't attempt to load images when they're deemed unavailable
+ * @return void
+ */
+ public function testImageSkippedWhenUnavailable() {
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+ $gd = new GDBackend_ImageUnavailable($fullPath);
+
+ /* Ensure no image resource is created if the image is unavailable */
+ $this->assertNull($gd->getImageResource());
+ }
+
+ /**
+ * Tests the integrity of the manipulation cache when an error occurs
+ * @return void
+ */
+ public function testCacheIntegrity() {
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+
+ try {
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth', 123));

Should have a $this->failure() to make sure an exception's thrown.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ $this->fail('GDBackend_Failure should throw an exception when setting image resource');
+ } catch (GDBackend_Failure_Exception $e) {
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+ $key = md5(implode('_', array($fullPath, filemtime($fullPath))));
+
+ $data = unserialize($cache->load($key));
+
+ $this->assertArrayHasKey('SetWidth|123', $data);
+ $this->assertTrue($data['SetWidth|123']);
+ }
+ }
+
+ /**
+ * Test that GD::failedResample() returns true for the current image
+ * manipulation only if it previously failed
+ * @return void
+ */
+ public function testFailedResample() {
+ $fullPath = realpath(dirname(__FILE__) . '/gdtest/test_jpg.jpg');
+
+ try {
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth-failed', 123));

Likewise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ $this->fail('GDBackend_Failure should throw an exception when setting image resource');
+ } catch (GDBackend_Failure_Exception $e) {
+ $gd = new GDBackend($fullPath, array('SetWidth', 123));
+ $this->assertTrue($gd->failedResample($fullPath, 'SetWidth-failed|123'));
+ $this->assertFalse($gd->failedResample($fullPath, 'SetWidth-not-failed|123'));
+ }
+ }
+
+}
+
+class GDBackend_ImageUnavailable extends GDBackend implements TestOnly {
+
+ public function imageAvailable($filename, $manipulation) {
+ return false;
+ }
+
+}
+
+class GDBackend_Failure extends GDBackend implements TestOnly {
+
+ public function setImageResource($resource) {
+ throw new GDBackend_Failure_Exception('GD failed to load image');
+ }
+
+}
+
+class GDBackend_Failure_Exception extends Exception {
+
+}
View
39 tests/model/GDImageTest.php
@@ -1,5 +1,7 @@
<?php
+
class GDImageTest extends ImageTest {
+
public function setUp() {
if(!extension_loaded("gd")) {
$this->markTestSkipped("The GD extension is required");
@@ -25,4 +27,41 @@ public function setUp() {
$file->write();
}
}
+
+ public function tearDown() {
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+ $cache->clean(Zend_Cache::CLEANING_MODE_ALL);
+ parent::tearDown();
+ }
+
+ /**
+ * Test that the cache of manipulation failures is cleared when deleting
+ * the image object
+ * @return void
+ */
+ public function testCacheCleaningOnDelete() {
+ $image = $this->objFromFixture('Image', 'imageWithTitle');
+ $cache = SS_Cache::factory('GDBackend_Manipulations');
+ $fullPath = $image->getFullPath();
+ $key = md5(implode('_', array($fullPath, filemtime($fullPath))));
+
+ try {
+ // Simluate a failed manipulation
+ $gdFailure = new GDBackend_Failure($fullPath, array('SetWidth', 123));

Likewise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
+ $this->fail('GDBackend_Failure should throw an exception when setting image resource');
+ } catch (GDBackend_Failure_Exception $e) {
+ // Check that the cache has stored the manipulation failure
+ $data = unserialize($cache->load($key));
+ $this->assertArrayHasKey('SetWidth|123', $data);
+ $this->assertTrue($data['SetWidth|123']);
+
+ // Delete the image object
+ $image->delete();
+
+ // Check that the cache has been removed
+ $data = unserialize($cache->load($key));
+ $this->assertFalse($data);
+ }
+ }
+
}
Something went wrong with that request. Please try again.