accept file-like objects in np.load memmap mode #3306
Conversation
I didn't find any information why it was not allowed originally, commit 05fa6e7 does not give details, maybe np.memmap did not work with file-likes then |
@stefanv My guess is that it was a fix for Python 3.3. It might not be needed if an integer return is checked for. |
You are testing with a bona-fide |
gzip file is handled by a different function in np.load, it never goes into open_memmap. |
We're not "checking for gzip". We're documenting the restrictions on valid input to the function. You can't memory map onto arbitrary file-like objects.
|
I'm not sure what you are trying to tell me. |
You can update the documentation and the code to allow memmap-able file objects but there still needs to be documentation and hopefully an early check for general non-memmap-able file-like objects. Probably you can check that The reason that it was originally restricted to string filenames is that when you are given a valid Note that "file-like object" is a much broader term in Python parlance than you seem to be thinking of. It refers to any Python object with a minimal complement of methods like |
@juliantaylor I'd like to get this in, @rkern suggestions look good to me. |
I'm not so sure anymore, allowing arbitrary file descriptors does allow for a lot of mistakes, and as rkern said you can get the filename from a valid open descriptor easily. The only real reason I see to allow descriptors is if you are at the system limit of allowed descriptors. |
I think a more informative error message would be useful however this goes. |
Being able to memmap file descriptors is handy in all sorts of cases. I'm On Sat, May 4, 2013 at 6:52 PM, Charles Harris notifications@github.comwrote:
|
memmapping an unnamed file is certainly useful, but this pull is about np.load which does not make sense for these files. |
Oh, right then. (Sorry, I've been at a workshop all week.)
|
I'm going to close this for now, if someone comes up with a good use case we can reopen. |
In our project we have a strong use case for allowing |
@juliantaylor Want to reopen this? |
accept file-like objects in open_memmap
closes gh-3143