Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

support for an tf.io.gfile.GFile like API #354

Closed
kpe opened this issue Dec 23, 2020 · 1 comment
Closed

support for an tf.io.gfile.GFile like API #354

kpe opened this issue Dec 23, 2020 · 1 comment
Assignees
Labels
api: storage Issues related to the googleapis/python-storage API. status: duplicate Duplicate. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.

Comments

@kpe
Copy link

kpe commented Dec 23, 2020

It would be nice to support a tf.io.gfile.GFile like API because it can be used as a drop-in replacement for the normal pythonic open() API to transparently handle both local and Google Cloud Storage files (i.e. by using the gs:// schema).

Is your feature request related to a problem? Please describe.
Well it is much more convinient to use a familiar API like:

filepath = "gs://my-bucket/gcs-file.txt" if True else "/tmp/local-file.txt"
with gcs.GFile(file_path) as f:
  print(f.read())

then considering anything like:

#
# how exactly should I read the content of gs://my-buckets/gcs-file.txt 
# - where's the docu?
# - should I read it all?
# - I just need to read a text file (from GCS)
#
from google.cloud import storage
from google.cloud.storage.blob import Blob
client = storage.Client()
blob = Blob.from_string("gs://my-buckets/gcs-file.txt")
blob.download_as_*** # ... agghhh.h.. I'll just give up

Describe the solution you'd like
The way acess to GCS buckets is handled in TF (i.e. tf.io.gfile.GFile) is just perfect, however if I only need GCS file access, I may not want to install all of tensorflow.

Describe alternatives you've considered
As an alternative one could try splitting tf.io.gfile.GFile into a standalone pip package.

I'm not sure if this is the right place for a feature request like that, but it would be nice if there is a some pyton ideomatic open() like API. And if it were there, I guess, it would be used 90% of the time.

@product-auto-label product-auto-label bot added the api: storage Issues related to the googleapis/python-storage API. label Dec 23, 2020
@yoshi-automation yoshi-automation added triage me I really want to be triaged. 🚨 This issue needs some love. labels Dec 24, 2020
@frankyn frankyn added type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design. and removed 🚨 This issue needs some love. triage me I really want to be triaged. labels Jan 5, 2021
@frankyn
Copy link
Member

frankyn commented Jan 5, 2021

Thanks @kpe, this is a duplicate of: #29

@andrewsg is working on a solution.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
api: storage Issues related to the googleapis/python-storage API. status: duplicate Duplicate. type: feature request ‘Nice-to-have’ improvement, new feature or different behavior or design.
Projects
None yet
Development

No branches or pull requests

3 participants