imgproxy ate all free space on drive

[homm]

I'm trying to process 7360 × 4912 pixels image and after each request, the free space on the system drive is reduced by 104 MB. This doesn't stop until all space is gone.

$ lsof -p 16119 |grep /tmp
imgproxy 16119 ubuntu    3u      REG  202,1 108557789   2190 /tmp/vips-1-1VW16Y.v (deleted)
imgproxy 16119 ubuntu    7u      REG  202,1 108557789   2847 /tmp/vips-2-ITW36Y.v (deleted)

By the way, 7360*4912*3 ≈ 104 MB.

[DarthSim]

That's, actually, not totally correct. libvips does use your drive space to store decoded images that are too big to be securely stored in RAM. As you noticed correctly, 7360*4912*3 ≈ 104 MB, and many popular image processing tools like ImageMagick would store this amount of data on the drive.
Anyway, libvips removes all unneeded decoded data, so it won't eat all your space.

If you allow processing such big images on the fly, you should know the cost of it. They will either use a lot of your memory or drive space.

[homm]

This makes sense, but there are some problems:

1. Those files are deleted from the file system after request, but as you can see from lsof output, they are still opened by the imgproxy process and that is why OS can't free this space. So yes, during my tests imgproxy actually ate 4.8G of free space and left 0 bytes.

2. Disk storage is very-very slow compared to memory even with SSD. There are already limits for maximum image resolution, so imgproxy shouldn't use more memory than defined by those limits. There are no reasons to store any data on the disk if it fits into memory. And if the data doesn't fit, OS already has mechanisms to use disk storage if necessary: swap.

3. If this storage should prevent from high memory usage, why after the end of the short benchmark (ab -c 10 -n 50, all requests are finished) imgproxy consumes 13% (0.94Gb) of memory?

[DarthSim]

1. It seems like libvips reuses temporary files, I don;t see any other explanation of this behavior for now. I'll continue digging into it.
   Anyway, I made some changes in imgproxy, and now it uses sequential access mode instead of random access in most cases. The sequential mode doesn't create a temporary file and saves some memory. The only problem is that sequential mode isn't compatible with smartcrop, so I have to use random access mode with it.

2. Disk storage is slow, for sure. But RAM is a more valuable thing. There are a lot of reasons to not use all the memory we have. There may be other processes that need RAM. Or there may be RAM limits for the process, and it'll be killed instead of swapping. So it's not such a bad idea to use your disk instead of eating all the memory. Sure, wasting the disk with temp files isn't nice, but again, if you allow processing such big files on the fly, there will be consequences.
   If you still want to use your memory, you can set env variable VIPS_DISC_THRESHOLD=1000MB, to libvips won't save images of less than 1000 Mb decompressed size to disk.

3. imgproxy and vips still need memory to download and process files. The garbage collector in Go doesn't instantly return memory to the operating system, it uses allocated memory for the new objects. I made some optimizations and enforced GC to return memory more often. Now it looks much better to me.

Btw, could you share the image which you use for testing?

[homm]

Btw, could you share the image which you use for testing?

Sure, here it is: https://en.wikipedia.org/wiki/File:0000140_Wat_Arun_01.jpg

[homm]

Results with the current master:

• No disk swapping
• ≈400% CPU load during the test (unlike the previous version with only ≈150% CPU load).
• 2 times more throughput
• only 320 megabytes memory usage after the same test

So, everything is definitely much better )

[DarthSim]

Nice, thanks for testing!