Skip to content

scipy.io.netcdf leaks file descriptors (Trac #1028) #1555

Closed
scipy-gitbot opened this Issue Apr 25, 2013 · 1 comment

2 participants

@scipy-gitbot

Original ticket http://projects.scipy.org/scipy/ticket/1028 on 2009-10-22 by trac user alexg, assigned to unknown.

I am using scipy.io.netcdf and I’m having a problem with it:
when I loop over multiple files it crashes:

EnvironmentError: [Errno 24] Too many open files

Here is what my loop looks like:

ctt=[]
for f in filelist:
        ncfile=netcdf_file(path+f,'r')
        ctt.append( ncfile.variables['Cloud_Top_Types'][:] )
        ncfile.close()

Since I am closing the files at each loop I don’t understand
why it crashes. So either I am doing something wrong or the
module close() doesn’t work properly. Can someone please give
me an opinion or help me fix the bug?

I tried this script on a Mac and on a Linux machine.
The problem is qualitatively the same even though quantitatively
different. I can open up to 120 files on a Mac and between 720
and 960 on a Linux before it crashes. This tends to prove that
it depends on some internal limit (like the one one can tweak
with “ulimit”) while it shouldn’t if the files were really
closed by close().

@loujine
loujine commented Aug 25, 2013

I just took a look at that issue: it seems using mmap creates new file descriptors of the netcdf file (one fd for each variable created with netcdf_variable, and those are not closed with netcdf_file.close()

So N+1 fd are created when loading an .nc file (for N variables) but only one is destroyed with netcdf_file.close()

@loujine loujine added a commit to loujine/scipy that referenced this issue Aug 25, 2013
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
2956315
@rgommers rgommers added a commit to rgommers/scipy that referenced this issue Dec 21, 2013
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
a132b8e
@rgommers rgommers added a commit that closed this issue Jan 25, 2014
@loujine loujine BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when
using mmap

Closes gh-1555
2dfffc7
@rgommers rgommers closed this in 2dfffc7 Jan 25, 2014
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Something went wrong with that request. Please try again.