New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
accept file-like objects in np.load memmap mode #3306
Conversation
I didn't find any information why it was not allowed originally, commit 05fa6e7 does not give details, maybe np.memmap did not work with file-likes then |
@stefanv My guess is that it was a fix for Python 3.3. It might not be needed if an integer return is checked for. |
You are testing with a bona-fide |
gzip file is handled by a different function in np.load, it never goes into open_memmap. |
We're not "checking for gzip". We're documenting the restrictions on valid input to the function. You can't memory map onto arbitrary file-like objects.
|
I'm not sure what you are trying to tell me. |
You can update the documentation and the code to allow memmap-able file objects but there still needs to be documentation and hopefully an early check for general non-memmap-able file-like objects. Probably you can check that The reason that it was originally restricted to string filenames is that when you are given a valid Note that "file-like object" is a much broader term in Python parlance than you seem to be thinking of. It refers to any Python object with a minimal complement of methods like |
@juliantaylor I'd like to get this in, @rkern suggestions look good to me. |
I'm not so sure anymore, allowing arbitrary file descriptors does allow for a lot of mistakes, and as rkern said you can get the filename from a valid open descriptor easily. The only real reason I see to allow descriptors is if you are at the system limit of allowed descriptors. |
I think a more informative error message would be useful however this goes. |
Being able to memmap file descriptors is handy in all sorts of cases. I'm On Sat, May 4, 2013 at 6:52 PM, Charles Harris notifications@github.comwrote:
|
memmapping an unnamed file is certainly useful, but this pull is about np.load which does not make sense for these files. |
Oh, right then. (Sorry, I've been at a workshop all week.)
|
I'm going to close this for now, if someone comes up with a good use case we can reopen. |
In our project we have a strong use case for allowing |
@juliantaylor Want to reopen this? |
accept file-like objects in open_memmap
closes gh-3143