Segmentation fault (core dumped) from pyvips

[munirhossain]

I'm trying to resize a large image to a smaller width with a limited memory available 256MB (due to AWS lambda configuration). I used the code like this:

import pyvips

image = pyvips.Image.new_from_file(src_file, access='sequential')
image = image.resize(0.8/1.0)
image.write_to_file(dst_file)

In my local machine I'm testing it by putting a limit on Python VM using code like:

resource.setrlimit(resource.RLIMIT_AS, (maxsize, hard))

when I set the maxsize to 256MB, I get Segmentation fault (core dumped) error. I suppose this is coming from the vips C library. So, I have two questions:

1. Why the memory issue in vips library (assuming there is one) is not raised as MemoryError for python to handle gracefully?
2. How can I use pyvips to resize a very big image with relatively small memory environment such as AWS lambda? [I can sacrifice CPU performance but I need the memory performance]

[jcupitt]

Hello again,

You can do a range of things to limit libvips memory use:

1. Don't use resize, use thumbnail instead. It's much faster, needs less memory, and the quality is better.
2. Disable the libvips operation cache. libvips caches the result of the last 1,000 operations, or 100MB of memory, whichever is smaller. Use pyvips.cache_set_max(0) somewhere near the start of your program.
3. Shrink the libvips thread pool. libvips has a pool of worker threads, sized to fill your machine, and each worker has a set of memory buffers. Image shinking is not that CPU intensive, so threads don't really help much anyway. Do export VIPS_CONCURRENCY=1 before running your program.

[jcupitt]

I tried this program:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.leak_set(True)

image = pyvips.Image.new_from_file(sys.argv[1], access='sequential')
image = image.resize(0.8 / 1.0)
image.write_to_file(sys.argv[2])

leak_set makes libvips run a little slower, but it will check for leaks and report high-water memory usage. With a 10k x 10k pixel RGB image I see:

john@kiwi:~/try$ ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 53.48 MB

If I change the program like this:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.cache_set_max(0)
pyvips.leak_set(True)

image = pyvips.Image.thumbnail(sys.argv[1], 8000)
image.write_to_file(sys.argv[2])

I see:

john@kiwi:~/try$ VIPS_CONCURRENCY=1 ./resize.py ~/pics/wtc.jpg x.jpg
memory: high-water mark 33.82 MB

That's running with ulimit -m 256000, which might be equivalent to your settings. I couldn't get it to crash.

[jcupitt]

Sorry, I never answered your first question.

Out of memory errors are generally impossible to handle gracefully. Suppose memory is full and a fault is raised. What is going to handle the error? If no more memory can be used, no fault handler can execute. Like almost all software, libvips treats memory as an infinite resource and relies on the environment to manage it.

libvips does check large single allocations (for example, loading a 10gb image into memory), but does not check small ones, the reasoning being that, if a small allocation fails, the process is as good as dead.

[munirhossain]

Thanks a lot. I tried your example code:

#!/usr/bin/python 

import sys
import pyvips 

pyvips.cache_set_max(0)
pyvips.leak_set(True)

image = pyvips.Image.thumbnail(sys.argv[1], 8000)
image.write_to_file(sys.argv[2])

but my output was like:

Traceback (most recent call last):
  File "resize.py", line 20, in <module>
    image = pyvips.Image.thumbnails(sys.argv[1], 800)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/vimage.py", line 128, in call_function
    return pyvips.Operation.call(name, *args, **kwargs)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/voperation.py", line 132, in call
    op = Operation.new_from_name(operation_name)
  File "/home/munirh/venv/local/lib/python2.7/site-packages/pyvips/voperation.py", line 60, in new_from_name
    raise Error('no such operation {0}'.format(operation_name))
pyvips.error.Error: no such operation thumbnails
  VipsOperation: class "thumbnails" not found

I checked that I had vipsthumbnail command available. So how to use it through python?
What would be the best documentation for these? I saw that thumbnail() is there for pyvips.Image class but not the resize() and vips_thumbnail() is there for libvips API documentation. But how to get all for pyvips?

And where should I use ulimit -m 256000 in python to limit memory?

[jcupitt]

  File "resize.py", line 20, in <module>
    image = pyvips.Image.thumbnails(sys.argv[1], 800)

It should be thumbnail, of course, not thumbnails, but I guess you fixed that.

Otherwise it's probably because you are using an old version of the libvips shared library. thumbnail was added in 8.4 I think.

ulimit is a shell command, so run that before you start python. It limits every process started from that shell.

[abhijitgujar86]

we are having same issue when creating large image files around 2 gb +

[jcupitt]

Hello @abhijitgujar86,

Please open a new issue and include as much information as you can about the crash. Ideally, a reproducible test case.