BUG: close intermediate file descriptor right after it is used in netcdf.py

[jinhyokh]

netcdf opens intermediate file descriptors and they are closed when the main file descriptor is closed. But this causes a too many open files error when the number of intermediate descriptors are too many. This patch closes the intermediate file descriptor right after it is used.

[rgommers]

The failures are segfaults on all Python versions. Haven't checked if it's related, but this is new.

[jinhyokh]

I guess I fixed the problem. Please review my patch.

[rgommers]

Doesn't this now cause a lot of extra data copying? I assume you have some large netcdf files, otherwise you wouldn't see this. Can you take one that does load without this fix and time opening it in read mode?

[jinhyokh]

Without copy(), segfaults occur. Why does data.flags.writeable become True with copy()?

[rgommers]

I mean take one that works with current scipy master, and compare with this patch.

Flags: http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.flags.html

[jinhyokh]

Now it sets writeable flag properly and passes the tests.

[jinhyokh]

Sorry I didn't pay attention to the details. Considering the inefficiency of copy(), I realized this is not a good way of solving the problem. It may be better simply to advertise not to use mmap for netcdf having too many variables, or pass mmap=False in netcdf_file().

A better idea may be to check the number of potential mmap() calls and, if it is greater than resource.getrlimit(resource.RLIMIT_NOFILE), change self.use_mmap to False. How can I count the number of opened file descriptors in Python?

Or, any better idea?

[pv]

It should be possible to use a single mmap, initialized when the file is opened,
and obtain the variables via something like value = self._mm[start_byte_pos:end_byte_pos].view(dtype).reshape(shape)

[pv]

Also: isn't netcdf.py actually pupynere? The upstream is still active, so we should in principle track it: https://bitbucket.org/robertodealmeida/pupynere/commits/all

[rgommers]

@pv it was, a very long time ago. Last sync by Roberto (Pupynere author) was in 5b02684. After that a bunch of fixes went into scipy, that I don't think we contributed back. I'll open a separate issue for re-syncing with upstream.

[rgommers]

Travis error is a timeout, looks OK for this patch now.

[coveralls]

Coverage remained the same when pulling 0b79ea0 on heoj:master into c81c978 on scipy:master.

[jinhyokh]

I further modified to use buffer as pv suggested. Memory efficiency improved significantly!

[coveralls]

Coverage remained the same when pulling 09d235e on heoj:master into c81c978 on scipy:master.

[betodealmeida]

There's another major improvement in using ALLOCATIONGRANULARITY when available. See:

https://bitbucket.org/robertodealmeida/pupynere/src/0fa832fa4400da818019f7d2883669c1ce08823f/pupynere.py?at=default#cl-93

When ALLOCATIONGRANULARITY is not set the mmap for each variable in the file is created from the start of the file to the end of the variable, since it's not possible to specify an offset; the offset is later specified when creating the Numpy array. For files with many variables this is highly inefficient. The code at bitbucket will check if ALLOCATIONGRANULARITY is available (on Python >= 2.6), and makes use of the paging.

[pv]

Merged with minor changes in 58d8117