Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Feature Request: Support Text IO #52

Closed
asford opened this issue Jan 2, 2018 · 2 comments
Closed

Feature Request: Support Text IO #52

asford opened this issue Jan 2, 2018 · 2 comments

Comments

@asford
Copy link
Collaborator

asford commented Jan 2, 2018

It would be valuable to support the standard Text I/O interface in gcsfs for use as a standalone, file-like library. The current behavior requires binary IO, which is not directly compatible with some standard tools (eg. json).

For example:

import json
import gcsfs

fs = gcsfs.GCSFileSystem(...)
an_obj = {"foo" : "bar"}

with fs.open("bucket/path", "w") as of:
    json.dump(an_obj, of)

raises NotImplementedError.

While it's possible to workaround via io.TextIOWrapper, it would be best to expose a GCSFile in text-mode in order to provide support for gcsfs-specific api extensions on the file object.

import json
import gcsfs

fs = gcsfs.GCSFileSystem(...)
an_obj = {"foo" : "bar"}

with fs.open("bucket/path", "wb') as of:
    json.dump(an_obj, io.TextIOWrapper(of))
@asford
Copy link
Collaborator Author

asford commented Jan 3, 2018

For clarity, particularly for anyone looking to use gcsfs alone, the dask.bytes.core.OpenFileCreator interface can be used to access this interface with under a TextIOWrapper.

@martindurant
Copy link
Member

Certainly, seem easy enough to implement should 'b' should not be in the mode, I guess within GCSFileSystem.open rather than GCSFile.__init__?
Note that there is already some similar logic in dask.bytes.core.OpenFile, used, for instance, by dak.bag.to_textfile.

martindurant pushed a commit to martindurant/gcsfs that referenced this issue Jan 9, 2018
Plus text mode (see fsspec#52)
martindurant added a commit that referenced this issue Jan 10, 2018
* First stab at using google.auth

Use new purpose-built package to generate sessions and handle refresh,
simplifying our code.

* Don't error if a connection method fails

* Re-enable token= access.

* Enable auth with Credentials

* Fixes

Plus text mode (see #52)

* Fix tests

(only three failures left)

* rerecord tests

* Better token hiding

* update dependencies for travis

* utf8 for py2 in test_core; drop py 3.4 on travis

(google requirememnts are not built for py3.4)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants