Capacity error while reading parquet file

[Ingvar-Y]

pyarrow.parquet.read_table fails while reading 2,69 GB parquet file written by Arrow with

ArrowIOError: Arrow error: Capacity error: BinaryArray cannot contain more than 2147483646 bytes, have 2147483647

This looks similar to #2485 and should have been resolved by ARROW-3762 but happens again in Arrow 0.13.0

[wesm]

Can you open a JIRA issue? If you have a reproducer that would be helpful. Does your file have nested types? e.g. list<string>

[Ingvar-Y]

Table metadata:

pyarrow.Table
id_1c: string
startdate_1c: timestamp[ns]
enddate_1c: string
removal_1c: uint8
type_1c: uint8
activation_1c: uint8
code_1c: uint16
product_1c: string
id_fin: uint32
id_doc: uint32
seller_1c: string
__index_level_0__: int64
metadata
--------
OrderedDict([(b'pandas',
              b'{"index_columns": ["__index_level_0__"], "column_indexes": ['
              b'{"name": null, "field_name": null, "pandas_type": "unicode",'
              b' "numpy_type": "object", "metadata": {"encoding": "UTF-8"}}]'
              b', "columns": [{"name": "id_1c", "field_name": "id_1c", "pand'
              b'as_type": "unicode", "numpy_type": "object", "metadata": nul'
              b'l}, {"name": "startdate_1c", "field_name": "startdate_1c", "'
              b'pandas_type": "datetime", "numpy_type": "datetime64[ns]", "m'
              b'etadata": null}, {"name": "enddate_1c", "field_name": "endda'
              b'te_1c", "pandas_type": "unicode", "numpy_type": "object", "m'
              b'etadata": null}, {"name": "removal_1c", "field_name": "remov'
              b'al_1c", "pandas_type": "uint8", "numpy_type": "uint8", "meta'
              b'data": null}, {"name": "type_1c", "field_name": "type_1c", "'
              b'pandas_type": "uint8", "numpy_type": "uint8", "metadata": nu'
              b'll}, {"name": "activation_1c", "field_name": "activation_1c"'
              b', "pandas_type": "uint8", "numpy_type": "uint8", "metadata":'
              b' null}, {"name": "code_1c", "field_name": "code_1c", "pandas'
              b'_type": "uint16", "numpy_type": "uint16", "metadata": null},'
              b' {"name": "product_1c", "field_name": "product_1c", "pandas_'
              b'type": "unicode", "numpy_type": "object", "metadata": null},'
              b' {"name": "id_fin", "field_name": "id_fin", "pandas_type": "'
              b'uint32", "numpy_type": "uint32", "metadata": null}, {"name":'
              b' "id_doc", "field_name": "id_doc", "pandas_type": "uint32", '
              b'"numpy_type": "uint32", "metadata": null}, {"name": "seller_'
              b'1c", "field_name": "seller_1c", "pandas_type": "unicode", "n'
              b'umpy_type": "object", "metadata": null}, {"name": null, "fie'
              b'ld_name": "__index_level_0__", "pandas_type": "int64", "nump'
              b'y_type": "int64", "metadata": null}], "creator": {"library":'
              b' "pyarrow", "version": "0.13.0"}, "pandas_version": null}')])

As you see, there is no nested types. Table has ~180m rows, all columns besides id_fin form a subtable with ~40m unique rows with id_1c and id_doc as key columns. id_1c contains UUID hex strings, enddate_1c mostly contains datetime data with some proper strings and ~15k unique values, product_1c contains strings up to 150 characters long and ~800 unique values, seller_1c contains strings up to 50 characters long and 12 unique values, id_fin has ~135m unique values.

I'll see if I can make a reproducer, the same structure worked for me with previous versions of source data, my best guess is that problems came from increasing volume.

@wesm Do I still need to open a JIRA issue since you reopened ARROW-3762?

[Ingvar-Y]

I'm not sure if it is related but there is some problem with creation of ChunkedArray. I discovered this while tryingt to reproduce BinaryArray issue

import uuid
import numpy as np
import pyarrow as pa

li = []
for i in range(100000000):
    li.append(uuid.uuid4().hex)
arr = np.array(li)
parr = pa.array(arr)

This fails with

ArrowCapacityError: Encoded string length exceeds maximum size (2GB)

I thought this was solved as per docs and #1855

[wesm]

Thanks. Closing and have reopened ARROW-3762 issue