-
-
Notifications
You must be signed in to change notification settings - Fork 385
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to read gz files on s3 #12
Comments
Transparent (de)compression is only supported for local files now. Should be possible to do it transparently for S3 files too, using Python's |
+1 for this issue, I have some example code here: https://gist.github.com/brianmingus/a47f26760d244ba7e9d1 |
It's interesting how smart_open is evolving into something similar for the Python world to what Apache Commons VFS is to the JVM world. Maybe there's some inspiration of the abstractions they used to build something more general. |
This is useful for me as well, as soon as #38 gets merged I could write a PR for this one as well. |
Thanks for the link @asieira -- I didn't know about VFS. I'm all for learning from other people's mistakes -- what abstractions and designs in particular do you think would be useful? |
The first thing they did was to create the abstraction for a file system, not opening a single file. Going the single file route as smart_open has gone so far is great for formats like xz, gzip and bzip... mas if you want to handle archives that themselves contain internal structure (like zip, tar, etc) that won't work as well. So in VFS, opening a ZIP file is akin to traversing a virtual folder to access the content inside. Plus, they worked in an arbitrary number of layers. You can build a URI like gz://zip://ftp://ftp.example.com/file/blah.zip!/zipfolder1/file.gz and access a GZIP file, inside a ZIP file, read from an FTP server. Plus, it's an architecture that can be extended. They defined a set of abstract classes that you can implement to define a new type of filesystem in addition to the built-in ones. All great ideas, but maybe too complicated for smart_open and worthy of a separate independent project that emulates those ideas in Python. At the very least, thinking of those ideas I would be inspired to implement support for gzip and bzip compression / decompression in a way that is general for all supported file types in smart_open. |
+1 on this, I might be able to implement it soon since we'll probably need it..thanks |
Sounds great @AndreaCrotti ... that would be really useful! |
This seems to work pretty well as a work-around: https://github.com/commoncrawl/gzipstream |
Sounds good, thanks for the link @mpenkov ! Can you implement this in a PR? Depending on how tricky gzipstream is to install and how well supported it is, we could either add it to requirements (if easy), or make it optional (if difficult), or even bundle it inside smart_open directly (license permitting). @tmylk great intro task? |
gzipstream doesn't really have any external requirements (just io and zlib) @piskvorky I could in theory, but I'm working on something else right now. On Fri, Jun 3, 2016 at 2:14 PM Radim Řehůřek notifications@github.com
|
@piskvorky OK, I'm looking into this now. |
- Bundle gzipstream to enable streaming of gzipped content from S3 - Update gzipstream to avoid deep recursion - Implement readline for S3 - Add pip requirements.txt
@piskvorky @tmylk I think we can close this now. 78c461e resolved this. |
Thanks @mpenkov ! Closing now |
It seems that smartopen can read a gzip file from s3 using an url, but not using key. Is that the case? |
It reads them, but it the data remains compressed, thus defeating line iteration.
The text was updated successfully, but these errors were encountered: