TensorFlow GCS access does not work from colab

[rsepassi]

System information
Using colab.research.google.com

Describe the current behavior

Hangs.

import tensorflow as tf
tf.io.gfile.exists(“gs://tfds-data”)  # which is a public GCS bucket

Describe the expected behavior

Should not hang.

[rsepassi]

Linked tensorflow/datasets#36

[craigcitro]

Yes, this one's a known bug: the problem is that it's trying an authenticated request, with some number of timeouts; IIRC, it'll eventually complete after ~10m. (!!!)

Quick workaround: first, do

from google.colab import auth
auth.authenticate_user()

and it'll work.

This was intended to be fixed upstream, but it looks like that didn't work (at least, it still hangs for me). I'll take a look.

[rsepassi]

Thanks @craigcitro. Why is it trying to make an authenticated request? In the particular case I'm running into, it's a publicly accessible GCS bucket, so no auth necessary. Where is the "authenticated request" logic happening? I'm guessing it's a TF thing, not a colab thing, right? Should we update it to first check if credentials exist? (if so, make authenticated request, if not don't)

In my particular case, TFDS is storing some files on a public GCS bucket and is trying to load them. The user shouldn't know anything about the GCS bucket. Calling auth.authenticate_user() forces the user to go to an oauth link, copy a verification code, and paste it back in. This isn't something we want to force users to do. In the short term TFDS will probably use the GCS HTTP API instead of tf.io.gfile.

[craigcitro]

I've added a $NO_GCE_CHECK environment variable that allows a user to completely sidestep the GCE metadata checks; anyone who's hitting the original issue (slow timeouts + retries attempting to fetch a public GCS resource) should be able to use this to get unblocked.

For Colab in particular, I'm going to add this patch and enable it in our runtimes (where GCE metadata is never available).

[rsepassi]

Thanks!

…

On Wed, Feb 6, 2019 at 10:24 PM Craig Citro ***@***.***> wrote: I've added a $NO_GCE_CHECK environment variable that allows a user to completely sidestep the GCE metadata checks; anyone who's hitting the original issue (slow timeouts + retries attempting to fetch a public GCS resource) should be able to use this to get unblocked. For Colab in particular, I'm going to add this patch and enable it in our runtimes (where GCE metadata is never available). — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#25463 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ABEGWxqzc1Y2eueoxJ6Cux6txS1tKe5Dks5vK8aIgaJpZM4af5x9> .