GCS support

[kentslaney]

While trying to load a TFDS dataset from GCS, I encountered this error:

/usr/local/lib/python3.10/dist-packages/array_record/python/array_record_data_source.py in _create_reader
    176 def _create_reader(filename: epath.PathLike):
    177   """Returns an ArrayRecordReader for the given filename."""
--> 178   return array_record_module.ArrayRecordReader(
    179       filename,
    180       options="readahead_buffer_size:0",

RuntimeError: open() failed: No such file or directory; opening gs://ref_coco/ref_coco/refcoco_unc/1.1.0/ref_coco-train.array_record-00000-of-00032

See the minimum reproducible example here:

https://colab.research.google.com/drive/1iezVDZBJrWtP3fVTpAMNGb6qfENZNKuD?usp=sharing

[kentslaney]

I should mention that I'm not blocked right now because of this, the dataset is small enough to fit on disk

[dryman]

GCS relies on upstream library riegeli. Maybe file a request over there and I can add it into array_record after riegeli supports it.

[666even666]

I encountered the same issue recently when loading imagenet from GCS. Imagenet would not fit on my local disk. Is there a suggested workaround? Thanks!