Using etils's epath with AWS S3 links

[neel04]

Hey,

Apparently, one can't use S3 objects with etils:

from etils import epath
a = epath.Path( 's3://....' ) #link points to a folder of .tfrecords
sorted(epath.Path('s3://..').iterdir())

Error

Yields,

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.8/dist-packages/etils/epath/gpath.py", line 126, in iterdir
    for f in self._backend.listdir(self._path_str):
  File "/usr/local/lib/python3.8/dist-packages/etils/epath/backend.py", line 191, in listdir
    return self.gfile.listdir(path)
  File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/lib/io/file_io.py", line 766, in list_directory_v2
    raise errors.NotFoundError(
tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory s3://..

This utilized in TFDS simply to compute some meta-data for the datasets I'd be working in.

I can confirm the given link exists when copypasted to the aws s3 ls s3://.... command. It seems an issue with etils, or I'm using it wrongly.

Cheers!

[Conchylicultor]

Could you try using TF directly to see if the issue is coming from TF or epath ?

import tensorflow as tf

tf.io.gfile.listdir('s3://...')

[neel04]

I assume epath, because the tracebacks are same

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/awesome/.local/lib/python3.7/site-packages/tensorflow/python/lib/io/file_io.py", line 769, in list_directory_v2
    message="Could not find directory {}".format(path))
tensorflow.python.framework.errors_impl.NotFoundError: Could not find directory s3://...

[Conchylicultor]

If you also get the same error using only tf.io.gfile (without epath), it means the error is coming from TensorFlow, not epath, right ? Or I have misunderstood something ?

If so, you should open a bug in TF with the TF example (without epath) which reproduce the issue.

[neel04]

Fair enough; I'd opened an issue with TF but they weren't helpful; ultimately resolved with downgrading.
I'm trying to use with TFDS but have been facing a barrage of errors with S3 😞 absolutely plagued by No response body, Trying for days to resolve this; kinda regretting using TF over PyTorch.

Thanks for your help though.