[BUG] Reading data from GCS creates issue

[bschifferer]

Describe the bug
Reading parquet file from Google Cloud Storage does not work.

Steps/Code to reproduce bug

dataset = nvt.Dataset("gs://bucket/file.parquet")
dataset.to_ddf().head()

Error:

cuDF failure at: ../src/table/table.cpp:42: Column size mismatch:

If the data is copied to the local disk, the code will work.
cuDF / dask_cudf can read from GCS.
This is with the latest NVTabular

[rjzamora]

@bschifferer - I am having trouble reproducing this. However, I did run into another error related to the local buffer needing to be wrapped in a BytesIO object in order to interact with pyarrow correctly. Perhaps the errors are related?

When you say "cuDF / dask_cudf can read from GCS", are you referring to the latest version of RAPIDS, or 21.08?

[pchandar]

@rjzamora

As @bschifferer said, this fails

dataset = nvt.Dataset("gs://bucket/file.parquet")
dataset.to_ddf().head()

but this seems to work fine (feel a lot slower)

ddf = dask_cudf.read_parquet("/path-to-data/*.parquet")
nvt.Dataset(ddf)

[rjzamora]

So this still doesn't work with the latest development branch (main)? Is the problematic file in a public bucket? If not, can you share a toy DataFrame example that cannot be read back like this after being written to gcs?

[pchandar]

I haven't tried with the latest main branch. I'll check if #1158 fixed the issue and provide an update.

[rjzamora]

I haven't tried with the latest main branch. I'll check if #1158 fixed the issue and provide an update.

Thank @pchandar ! Note that I am not very confident that #1158 was related, but since I cannot reproduce with an arbitrary parquet file myself, this issue is a bit difficult to debug.

[pchandar]

@rjzamora sorry it took a while. It was a bit tricky to reproduce this on a test dataset. But if you copy the transformed parquet from this (Cell 18) example to a GCS bucket and then

ds = nvt.Dataset("gs://bucket/movielens.parquet")
ds.head()

will give the following error

RuntimeError: cuDF failure at: ../src/table/table.cpp:42: Column size mismatch: 76 != 20000076

A couple of observations: (1) this seem to happen only when the list columns exists; and (2) for sufficiently large datasets (when I tried slicing the problematic dataset it seemed to work fine). Hope this help reproduce the error at your end.
Thanks

[rjzamora]

I'm sorry for the delay here. This is indeed a bug in the optimized data-transfer logic for read_parquet from remote storage. It turns out that that the list column name is modified from "genres" to "genres.list.element" in the parquet metadata, and so we fail to transfer the data for that column. In the near future, all this logic will live directly in fsspec (and will be removed from NVTabular), but I will submit a temporary fix asap for NVT.