Original ticket http://projects.scipy.org/scipy/ticket/1028 on 2009-10-22 by trac user alexg, assigned to unknown.
I am using scipy.io.netcdf and I’m having a problem with it:
when I loop over multiple files it crashes:
EnvironmentError: [Errno 24] Too many open files
Here is what my loop looks like:
for f in filelist:
ctt.append( ncfile.variables['Cloud_Top_Types'][:] )
Since I am closing the files at each loop I don’t understand
why it crashes. So either I am doing something wrong or the
module close() doesn’t work properly. Can someone please give
me an opinion or help me fix the bug?
I tried this script on a Mac and on a Linux machine.
The problem is qualitatively the same even though quantitatively
different. I can open up to 120 files on a Mac and between 720
and 960 on a Linux before it crashes. This tends to prove that
it depends on some internal limit (like the one one can tweak
with “ulimit”) while it shouldn’t if the files were really
closed by close().
I just took a look at that issue: it seems using mmap creates new file descriptors of the netcdf file (one fd for each variable created with netcdf_variable, and those are not closed with netcdf_file.close()
So N+1 fd are created when loading an .nc file (for N variables) but only one is destroyed with netcdf_file.close()
BUG io: fix file descriptor closing for netcdf variables
File descriptors created for each Variable stayed open when