New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
coders/pnm: Handle write failures #2081
Conversation
From a quick search, this seems to be a fairly common error:
|
Confirmed the bmp writer has the same problem. I can try to create some resuable macros, since most What do you think? Something like this, but better, of course:
|
Example output would then be:
|
It seems that we ignore the return value in a lot of places. But we never use #define NewWriteStringMacro(buffer)
{
size_t
write_length;
write_length=strlen(string);
if (WriteBlobStream(image,write_length,(const unsigned char *) buffer) != (ssize_t) write_length)
status=MagickFalse;
} And then check |
Thank you for your response. I'm toying with some macros right now and will convert Understood, regarding perror. |
It might be wise to limit your actions to the p.s. We have strict code style rules. Could you please take a look at the style of the current code and take the style in account in you purposal? |
We would prefer to not use a macro in this situation and solve it inline instead. Similar to what you have done now. p.s. It looks like you could re-use |
Without a macro, code like this might get quite convoluted: https://github.com/ImageMagick/ImageMagick/blob/master/coders/bmp.c#L2289 https://github.com/ImageMagick/ImageMagick/blob/master/coders/bmp.c#L2345 I'll push an updated version for pnm and bmp to this PR (no need to merge it as is, seems like there's useful discussion going on), and then please let me know what you think makes most sense. |
7ab5cf1
to
c9bebc4
Compare
Your comment about the bmp coder gave me another idea. Would it not be sufficient enough to only check the last write? If there is no more diskspace all of them will fail. |
TODO: - Get rid of perror() - Pick proper function name for each macro (not literal WriteBlob*)
Updated the current branch (not ready for merging at all) Couple notes:
For code that does actually check the return values, the macro cannot be used, obviously. |
How would you handle:
|
Here is the recommended practice for addressing detecting I/O problems:
|
Ok, that works:
Will add that to the macro instead of perror and fprintf. |
We would prefer you to not use a macro to cover the whole situation. There are just too many differences between the coders. And t is possible that you will need to clean up memory before you do a break or return. And to answer your question. You could do this: status=CheckWriteBlobLSBLong(image,bmp_info.image_size);
status&=CheckWriteBlobLSBLong(image,bmp_info.x_pixels);
status&=CheckWriteBlobLSBLong(image,bmp_info.y_pixels);
status&=CheckWriteBlobLSBLong(image,bmp_info.number_colors);
status&=CheckWriteBlobLSBLong(image,bmp_info.colors_important);
if (status == MagickFalse)
{
ThrowFileException(exception,FileOpenError,"UnableToWriteFile",
image->filename);
// Do things
} And |
c9bebc4
to
44a74ec
Compare
This looks like a good approach me. EDIT: Actually, no, this has problems, since |
Its also not necessary to check every write. Recommended practice is to have the single check in the loop when pixels are written. That will capture any I/O errors that occur. |
Previously the code would always just return MagickTrue, even if it had just inferred that the processing failed. This is required for more elegant error handling throughout the function, such as checks on the WriteBlob() calls, where the return value is currently being ignored. Upon WriteBlob errors, we can then set the status to MagickFalse, and break.
In WritePNMImage, when converting to a PNM image, the result of the WriteBlob calls is not checked, and instead ignored explicitly by '(void)' in front of it. This means that ImageMagick return success on images that get truncated when write(2) fails with ENOSPC. Evident by this sequence: $ ls -lsh /tmp/0009-1-small.png 20M -rw-r--r-- 1 merlijn merlijn 20M May 25 23:59 /tmp/0009-1-small.png $ sudo mkdir /mnt/test $ sudo mount -t tmpfs none -o size=1G /mnt/test/ $ df -h /mnt/test/ Filesystem Size Used Avail Use% Mounted on none 5.0M 0 5.0M 0% /mnt/test $ convert /tmp/0009-1-small.png /mnt/test/0009.pnm $ echo $? 0 $ ls -lsh /mnt/test/0009.pnm 5.0M -rw-r--r-- 1 merlijn merlijn 5.0M May 31 11:42 /mnt/test/0009.pnm Failure is evident with strace: write(5, "\373\372\367\373\371\367\372\370\365\372\370\364\372\370\364\372\371\366\372\367\363\363\355\346\351\344\327\370\363\356\372\370"..., 8192) = -1 ENOSPC (No space left on device) MagickCore/blob.c handles these errors nicely, but the callers of WriteBlob should still check the actual amount of written bytes. Since MagickCore/blob.c already handles EINTR for them, the checks are simple. This commit only fixes up the PNM image writing, but similar issues remain throughout the file.
How would this work in the following scenario:
Now we'll end up with an image without a proper header, and imagemagick will still return success. That seems undesirable to me. |
Add minimal checks as we recommended in the coders to reduce the exception footprint in the coders. It will capture all/most of IO exceptions. For your proposed scenario, we already capture that in CloseBlob() with ferror(). All we need to do is to check the status of CloseBlob() and throw an exception if its not MagickFalse. Something like
|
Hm... If ferror was supposed to work, then why does the current code not print any errors? ... I suppose that for that to work, the return value of Unless you want to check for ferror again in WriteImage? |
As our use case, check CloseBlob() in coders/pnm.c. That should capture all I/O errors:
|
This feels fragile. It only works if everything uses It looks like the blob code also uses |
More specifically, if WriteBlob relies on anything that uses anything other than |
CloseBlob() check the blob status. If it returns MagickFalse, it has an error from a file, pipe, stdin, etc. If any of these fail, an exception will be throw per the recommended practice:
We can do the same check for the readers with the AnErrorHasOccurredReadingFromFile exception severity. |
So, as far as I can see, for anything that is not directly under your control, you'd have to explicitly check if |
Rather than speculate, we would need a use case so we can reproduce the problem. If a problem exists, we might be able to fix the problem within blob.c. Of course, you can check each write call as @dlemstra suggests: status&=Write..., to be complete, but its likely unnecessary. We have a catchall with checking the status of CloseBlob() until proven otherwise and then we fix any issues that we missed on a case by case basis. |
The So, assuming that:
I can rework my PR to add the inline functions to |
Recall, you can choose to instead check every write. Consider what @dlemstra and @urban-warrior offered and converge on a final pull request. We'll squash and merge it and if it proves to be an optimal soluton, we can then clone it to the other coders. |
I'll try to update this coming weekend at the latest and check the status of each call, thanks for the input. If I find the time I will also make an example with the gz |
I came up with one more idea, which I actually quite like. Define/add a struct that can be passed to an (inline) wrapper around the various
SafeWriteBlob would be something like:
This has the added advantage that we can store errno on the first error, and preserve it, until CheckBlobStatus is called. In the extreme, you could call it only once you're done with a file (just before And of course, if you really like the idea API, you could integrate it into the blob API somehow. When I find some time I'll try to update my PR to this, if you like the idea. |
ImageMagick already tracks each exception that is thrown. That is, ImageMagick does not replace an exception when a new one is thrown. Instead it keeps a list. We can replicate your idea simply by throwing additional exceptions inside of the Blob methods. Right now we return a status. Instead, we can issue warning/error exception and when the exceptions are revealed to the user, a list of all I/O exceptions are included. Any exception can be caught by checking the blob method status and/or checking exception->severity > warning. Any blob method that include the Image* parameter can directly throw an exception. For the low-level methods that do not include an Image*, we would need to add an ExceptionInfo structure to track exceptions and push them to the Image* parameter of the GetBlobError() method. CheckBlobStatus() is not required. You can do that now with the existing GetBlobError(). We like your ideas and we do need to robustly address possible I/O exceptions, but we certainly want to leverage existing functionality and apply the KISS principle in addressing the issue, whenever possible. If you agree with adding exceptions to the blob methods, let us know. We can code up that solution and you can then leverage the functionality in your PNM patches. |
Yeah, using existing mechanisms seems fine. KISS is fine. So then you would do what my suggested And then the recommended way to use this functions would to:
I can imagine you don't want to have a list with >100000 exceptions because every write on a large image fails, so that's probably good to keep in mind. (The single errno + status bit doesn't have this problem, fwiw) If you can create an example with the exceptions in the blob methods, I'd be happy to leverage them to the PNM coder and see how it goes. |
Ok, here's the plan. We'll will add exception handling insider of the Blob handler by the end of this weekend. We'll add one check only in coders/pnm.c to show the promise of this solution. We will then want feedback from you and @dlemstra to finalize our approach. Once we're all in agreement, you can finalize your patches to the PNM coder. Once reviewed and approved, we export your model to the other coders. Thanks for the discussion. Its important to get this right before we unleash our changes to over 130 coders modules. Note, the exception handler limits the # of exceptions to 1000 to help prevent excessive reporting. |
Sounds good, thanks for being patient with me & replying quickly! |
The proper place for catching I/O exceptions is in the blob methods. Our patch resolves most if not all potential I/O exceptions within MagickCore/blob.c such that little or no patching is required in any of the coders. Although we could check the return value of each read/write, it is not necessary as any I/O exception is now noted via errno when GetBlobError() is called. We will push the patch, likely sometime tomorrow. With the patch, we get expected results:
|
Ok, I see the commits made it in the master branch. I'll update this PR this weekend - thanks! |
2020-06-12 7.0.10-19 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-19, GIT revision 17343:e552d22:20200612 2020-06-09 7.0.10-19 Cristy <quetzlzacatenango@image...> * Improve checking for write failures (reference ImageMagick/ImageMagick#2081). 2020-06-08 7.0.10-18 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-18, GIT revision 17333:d071c2032:20200608 2020-06-08 7.0.10-18 Cristy <quetzlzacatenango@image...> * Colorspace change will remove ICC profile (reference ImageMagick/ImageMagick6#82). 2020-06-07 7.0.10-17 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-17, GIT revision 17311:8b5350f:20200607 2020-06-03 7.0.10-17 Cristy <quetzlzacatenango@image...> * Free up memory after a ICC profile is removed. 2020-05-31 7.0.10-16 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-16, GIT revision 17294:5be1abe:20200531 2020-05-30 7.0.10-16 Cristy <quetzlzacatenango@image...> * Fix PDF XREF directory for image sequences with and without ICC profiles. 2020-05-29 7.0.10-15 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-15, GIT revision 17282:9294896:20200529 2020-05-24 7.0.10-15 Cristy <quetzlzacatenango@image...> * Clipping was not returning expected results (reference ImageMagick/ImageMagick#2061). * Don't write a ICC profile to PDF if the image is gray (reference ImageMagick/ImageMagick#2070). 2020-05-22 7.0.10-14 Cristy <quetzlzacatenango@image...> * Release ImageMagick version 7.0.10-14, GIT revision 17268:e9c804c93:20200522 2020-05-22 7.0.10-14 Cristy <quetzlzacatenango@image...> * Errant warning when reading a profile file (reference ImageMagick/ImageMagick#2030). * Fix one off error on PDF object for images with ICC profile.
Sorry, I was quite busy. I checked out the current work, and it seems to resolve the problems:
Thanks! |
Prerequisites
Description
When using
convert
, coders/pnm fails to detect is the disk is full, and instead return exit code0
. TheWrite*
calls throughout the file do not check the actual amount of bytes that are written, leading to silent truncation errors. Additionally, the actual return line at the end of the function also unconditionally returns true, even if it has just detected that the processing went wrong.Further description, including how to reproduce, can be found in the commit messages, but here's a quick summary:
Please let me know if you are OK with this assessment and approach and I will fix up the rest of
coders/pnm.c
in this pull request, and briefly scan some of the others coders as well.