Preventing ImageMagick from leaving temp files

[mertalev]

It seems that when magick serializes files to disk, there are cases where these files won't get removed either by magick or by libvips. This can swell disk usage quite a bit: the issue here describes /tmp having 43GB worth of magick files. Based on the logs in that issue, it might be that an error in libvips results in these files getting left behind.

This is unfortunately difficult to reproduce. My attempt was to use the env variables like MAGICK_MEMORY_LIMIT and MAGICK_AREA_LIMIT here to force it to serialize to disk and pass an invalid TIFF file to throw an error simultaneously, but it hasn't worked yet. Not sure if it's that libmagick ignores those envs, the timing is off, or if it's that the issue is actually something else.

If I know the envs work, I can prevent ImageMagick from serializing altogether by setting MAGICK_MEMORY_LIMIT infinite and MAGICK_MAP_LIMIT and MAGICK_DISK_LIMIT to 0, but I don't know if they're getting used. Do you have any suggestions for what I can do here?

[jcupitt]

Hi @mertalev,

Ouch that's bad.

Magick will go via disc for some delegate formats. The obvious ones are PDF (delegated to ghostscript), SVG (delegated to inkscape), and raw camera formats (usually delegated to ufraw I think). There are probably others.

It handles these formats by shelling out to to eg. ghostscript to render the file to PNM in /tmp, then loading the PNMs back in as a bitmap to be returned to the caller. Obviously shelling out to an external program and passing the image back in via a temp file is pretty inefficient, which is one of the main reasons libvips magickload is not really suitable for production use.

I think I would just turn off magickload and try to find out what formats users are uploading that need it. I'd guess raw camera files, but of course that's just a guess. Maybe there's some other way to support these images?

[mertalev]

Yep, it's just for RAW files with ImageMagick as a proxy for libraw. I didn't realize it writes to /tmp to pass the image to libraw 😅. Maybe I should start looking at alternatives to this setup. It's nice to have a single entry-point for image processing with the sharp API, but it seems like it's just moving complexity elsewhere.

[jcupitt]

libvips 8.13 added a system to let you block operations at runtime:

https://www.libvips.org/2022/05/28/What's-new-in-8.13.html

If you set VIPS_BLOCK_UNTRUSTED, it'll stop all unfuzzed loaders running, including magickload. I think I would recommend turning the flag on by default.

[jcupitt]

I had a quick look at immich. Nice job!

There's a new chapter in the docs for 8.16 with some suggestions for any devs working with libvips:

https://github.com/libvips/libvips/blob/master/doc/Developer-checklist.md

It might be worth a quick look.

[mertalev]

Thanks, that's really useful! I think there are some inefficiencies with how we use libvips right now. e.g. I think we crop to a buffer and pass that in again to resize. I'll take a look at refactoring that. Also, we use mimalloc with LD_PRELOAD. Is that fine or is there something about jemalloc that's better for libvips?

[jcupitt]

jemalloc is really good for highly threaded server processes with lots of malloc pressure. It can keep long-running processes from slowly fragmenting.

I don't know much about mimalloc.