unix error: Too many open files

[hackermd]

When I load images in Python via Vips.Image.new_from_file() I get the following error message when I'm trying to load the 254th image (the first 253 images are loaded without problems):

File "/Users/Markus/anaconda/lib/python2.7/site-packages/gi/overrides/Vips.py", line 398, in vips_image_new_from_file
gi.overrides.Vips.Error: No known loader for "TIFF/150117MPcycle_01_s0254_r12_c11_RFP_C02.png".
vips__file_open_read: unable to open file "TIFF/150117MPcycle_01_s0254_r12_c11_RFP_C02.png" for reading
unix error: Too many open files
VipsForeignLoad: "TIFF/150117MPcycle_01_s0254_r12_c11_RFP_C02.png" is not a known file format

I have already tried to increase the maximum number of open files on my system (OSX 10.9.5) and also via Vips.cache_set_max_files() - without success.

Weirdly, my colleague doesn't get the error although he uses the exact same code, same set of images and pretty much the same system configuration (OSX).

[jcupitt]

Most unixes have a limit on the number of open files per process, usually about 1,000. libvips needs two file descriptors per open file, plus a few others, so in practice you can't open more than about 400. The next version (due in April) needs only one per file, so you can open up to about 900 images.

Vips.cache_set_max_files() sets the number of file descriptors the operation cache can use. So if your main thread is running out, you actually need to lower this number (confusingly). Try setting it very low, 10 or so.

You could try the git master version. It passes the test suits, so it ought to be as stable as the official supported version.

I work with PET scans which can have up to about 5,000 images. I have to process these in sections: I make a first pass to assemble a set of frames to disc, then a second pass on the assembled frames to make the final volume. I don't know if a trick like that could help.

[hackermd]

updated to Yosemite and now it works. weird! thank you for your quick feedback and advice!

[leandervaneekelen]

@jcupitt Hi John, sorry to necro an issue and asking you a question that's only tangentially related to pyvips, but I'm running into this exact error. I wanted to solve this by increasing the unix open file limit, but for whatever reason that's not working.

n_tiles_x, n_tiles_y = (...)
tiles = [...] # 2460 tiles @ (2048,2048) each
tiles_image = [
    pyvips.Image.new_from_file(
        tile
        access="sequential",
    )
    for tile in tiles
]
joined_image = pyvips.Image.arrayjoin(tiles_image, across=n_tiles_x)

This results in

unable to call VipsForeignLoadRadFile
/path/to/tiles/tile_10_67.v: unable to open for read
unix error: Too many open files

I tried setting ulimit -n <really_big_number> in the same terminal as where I'm doing my python calls and setting the user limits in /etc/security/limits.conf, but to no avail. I've had the most luck with running

import resource as res
_, hard = res.getrlimit(res.RLIMIT_NOFILE)
res.setrlimit(res.RLIMIT_NOFILE, (hard, hard))

# Do array join here...

but this only seems to work when all code is within the same scope/script and breaks the moment I put my code across multiple files/modules, suggesting the subprocesses that spawn don't inherit this set constant. Any suggestions for dealing with this?

[jcupitt]

Hey @leandervaneekelen, you'll need to check your distro, they are all slightly and annoyingly different. You might need to reconfigure your kernel, even.

[leandervaneekelen]

Hi @jcupitt, for posterity's sake: in the end I couldn't work out how to set the file limit, so I simply increased my tile sizes to a point where I was below this limit. Admittedly I have the luxury of being able to deal with the corresponding need in memory by simply reserving a bigger job on our HPC cluster, so this is a less-than-elegant solution for other parties probably.

[jcupitt]

HPC clusters are complicated, you'd need to talk to your sysadmin about raising the files limit, if it's even possible.

suggesting the subprocesses that spawn don't inherit this set constant.

If you're making a real subprocess, it might be on a different cluster node, and you'll need to raise the limit again. You could try pasting that rlimit code into every file.