Skip to content

HTTPS clone URL

Subversion checkout URL

You can clone with
or
.
Download ZIP

Loading…

scipy.io.netcdf leaks file descriptors (Trac #1028) #1555

Closed
scipy-gitbot opened this Issue · 1 comment

2 participants

@scipy-gitbot

Original ticket http://projects.scipy.org/scipy/ticket/1028 on 2009-10-22 by trac user alexg, assigned to unknown.

I am using scipy.io.netcdf and I’m having a problem with it:
when I loop over multiple files it crashes:

EnvironmentError: [Errno 24] Too many open files

Here is what my loop looks like:

ctt=[]
for f in filelist:
        ncfile=netcdf_file(path+f,'r')
        ctt.append( ncfile.variables['Cloud_Top_Types'][:] )
        ncfile.close()

Since I am closing the files at each loop I don’t understand
why it crashes. So either I am doing something wrong or the
module close() doesn’t work properly. Can someone please give
me an opinion or help me fix the bug?

I tried this script on a Mac and on a Linux machine.
The problem is qualitatively the same even though quantitatively
different. I can open up to 120 files on a Mac and between 720
and 960 on a Linux before it crashes. This tends to prove that
it depends on some internal limit (like the one one can tweak
with “ulimit”) while it shouldn’t if the files were really
closed by close().

@loujine

I just took a look at that issue: it seems using mmap creates new file descriptors of the netcdf file (one fd for each variable created with netcdf_variable, and those are not closed with netcdf_file.close()

So N+1 fd are created when loading an .nc file (for N variables) but only one is destroyed with netcdf_file.close()

@loujine loujine referenced this issue from a commit in loujine/scipy
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
2956315
@rgommers rgommers referenced this issue from a commit in rgommers/scipy
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
a132b8e
@rgommers rgommers closed this issue from a commit
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
2dfffc7
@rgommers rgommers closed this in 2dfffc7
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.