[Python][Parquet] Failure when reading Parquet file from S3 with s3fs

[asfimport]

I am reading parquet data from S3 and get  ArrowIOError error.

Size of the data: 32 part files 90 MB each (3GB approx)

Number of records: Approx 100M

Code Snippet:

from s3fs import S3FileSystem
import pyarrow.parquet as pq

s3 = S3FileSystem()

dataset = pq.ParquetDataset("s3://location", filesystem=s3)

df = dataset.read_pandas().to_pandas()

Stack Trace:

df = dataset.read_pandas().to_pandas()
File "/root/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1113, in read_pandas
return self.read(use_pandas_metadata=True, **kwargs)
File "/root/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 1085, in read
use_pandas_metadata=use_pandas_metadata)
File "/root/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 583, in read
table = reader.read(**options)
File "/root/.local/lib/python3.6/site-packages/pyarrow/parquet.py", line 216, in read
use_threads=use_threads)
File "pyarrow/_parquet.pyx", line 1086, in pyarrow._parquet.ParquetReader.read_all
File "pyarrow/error.pxi", line 87, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unexpected end of stream: Page was smaller (197092) than expected (263929)

 

Note: Same code works on relatively smaller dataset (approx < 50M records) 

 

 

Reporter: Siddharth
Assignee: Wes McKinney / @wesm

PRs and other links:

• GitHub Pull Request #5137
• Upstream s3fs issue

_{Note: This issue was originally created as ARROW-6058. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Seems like one of the files is corrupted. Have you tried reading the 32 files individually to see which one it is?

[asfimport]

Siddharth:
hey @wesm thanks for your prompt reply. I checked each part file and found the part file which gives an error. But the entire partition can be ready by redshift and athena without any errors. I can do a count(*) on the partition and it works so I am not sure if the part file is corrupted. But I can dig in to see if there is an issue with the part file. Any idea how we can fix this if part file was produced as expected?

 

 

[asfimport]

Andrey Krivonogov:
Hi @wesm,

I have experienced the same issue as [~sid88in]

I also managed to reproduce it with synthetic data:

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq
import s3fs

table = pa.Table.from_arrays([pa.array(np.arange(3 * 10 ** 7), type=pa.int64())], ['col'])
path = 's3://bucket/path/0.parquet'
fs = s3fs.S3FileSystem()
pq.write_table(table, path, filesystem=fs, row_group_size=10 ** 7)
table_read = pq.read_table(path, filesystem=fs)

 this snippet raises similar 

ArrowIOError: Unexpected end of stream: Page was smaller (241959) than expected (524605)

The problem seemed to be in s3fs version. Package versions I have

python 3.6.7
packages installed with conda (via conda-forge)

boto3==1.9.204
botocore==1.12.204
numpy==1.16.2
pyarrow==0.14.1

and it raised with

s3fs==0.3.3

but everything worked fine with

s3fs==0.2.2

 

Thank you in advance for your help !

 

[asfimport]

Wong Chung Hoi:
Hi all, 

FYI, I witness the same issue on BOTH GCP (with pandas.read_parquet and gcsfs) and AWS (pandas.read_parquet and s3fs).

 

I have also tried running the same code on the same dataset with an older docker build with older version of pyarrow and it works.

 

This is disabling us from using latest pyarrow to handle big parquet files.

[asfimport]

Wes McKinney / @wesm:
So far we don't have a minimal reproduction of the issue so it's very hard for other developers in this project to help. Since you are encountering the problem, you are the best positioned to reproduce the issue or determine the root cause.

[asfimport]

Wong Chung Hoi:
Hi all,

below is a simple piece of code to reproduce the issue using:

 

s3fs==0.3.3
pyarrow==0.14.1
pandas==0.24.0 

 

The file generated is roughly 170MB

 

import pandas as pd
import numpy as np
pd.DataFrame(np.random.randint(0, 10000, (10000000, 10)), columns=[str(i) for i in range(10)]).to_parquet('s3://path/to/file.snappy.parquet')
pd.read_parquet('s3://path/to/file.snappy.parquet')

Traceback (most recent call last):
 File "<stdin>", line 1, in <module>
 File "/Users/hoi/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/io/parquet.py", line 282, in read_parquet
 return impl.read(path, columns=columns, **kwargs)
 File "/Users/hoi/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pandas/io/parquet.py", line 129, in read
 **kwargs).to_pandas()
 File "/Users/hoi/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pyarrow/parquet.py", line 1216, in read_table
 use_pandas_metadata=use_pandas_metadata)
 File "/Users/hoi/.pyenv/versions/3.7.2/lib/python3.7/site-packages/pyarrow/parquet.py", line 216, in read
 use_threads=use_threads)
 File "pyarrow/_parquet.pyx", line 1086, in pyarrow._parquet.ParquetReader.read_all
 File "pyarrow/error.pxi", line 87, in pyarrow.lib.check_status
 pyarrow.lib.ArrowIOError: Unexpected end of stream: Page was smaller (304272) than expected (979599)

 

[asfimport]

Wes McKinney / @wesm:
Thank you, that's great! I added to the 0.15.0 milestone. I've been working a lot on Parquet stuff lately so if no one looks at it first I'll try to look before the release horizon closes

[asfimport]

Wes McKinney / @wesm:
I'm able to reproduce the issue. Going to dig in to see if I can find out what's going wrong with s3fs

[asfimport]

Wes McKinney / @wesm:
It's a bug in s3fs. I found a minimal reproduction and reported here fsspec/s3fs#220

[asfimport]

Wes McKinney / @wesm:
Issue resolved by pull request 5137
#5137

[asfimport]

Wes McKinney / @wesm:
Seems this can be fixed now by upgrading to fsspec==0.4.2