Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scipy.io.netcdf leaks file descriptors (Trac #1028) #1555

Closed
scipy-gitbot opened this issue Apr 25, 2013 · 1 comment
Closed

scipy.io.netcdf leaks file descriptors (Trac #1028) #1555

scipy-gitbot opened this issue Apr 25, 2013 · 1 comment
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected Migrated from Trac scipy.io
Milestone

Comments

@scipy-gitbot
Copy link

Original ticket http://projects.scipy.org/scipy/ticket/1028 on 2009-10-22 by trac user alexg, assigned to unknown.

I am using scipy.io.netcdf and I’m having a problem with it:
when I loop over multiple files it crashes:

EnvironmentError: [Errno 24] Too many open files

Here is what my loop looks like:

ctt=[]
for f in filelist:
        ncfile=netcdf_file(path+f,'r')
        ctt.append( ncfile.variables['Cloud_Top_Types'][:] )
        ncfile.close()

Since I am closing the files at each loop I don’t understand
why it crashes. So either I am doing something wrong or the
module close() doesn’t work properly. Can someone please give
me an opinion or help me fix the bug?

I tried this script on a Mac and on a Linux machine.
The problem is qualitatively the same even though quantitatively
different. I can open up to 120 files on a Mac and between 720
and 960 on a Linux before it crashes. This tends to prove that
it depends on some internal limit (like the one one can tweak
with “ulimit”) while it shouldn’t if the files were really
closed by close().

@loujine
Copy link

loujine commented Aug 25, 2013

I just took a look at that issue: it seems using mmap creates new file descriptors of the netcdf file (one fd for each variable created with netcdf_variable, and those are not closed with netcdf_file.close()

So N+1 fd are created when loading an .nc file (for N variables) but only one is destroyed with netcdf_file.close()

rgommers pushed a commit to rgommers/scipy that referenced this issue Dec 21, 2013
File descriptors created for each Variable stayed open when
using mmap

Closes scipygh-1555
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
defect A clear bug or issue that prevents SciPy from being installed or used as expected Migrated from Trac scipy.io
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants