Join GitHub today
GitHub is home to over 28 million developers working together to host and review code, manage projects, and build software together.Sign up
ENH: Parquet, HDF5, and CSV interfaces #1194
@wesm is it better map parquet logical types directly to ibis types, or convert to arrow types which then almost trivially map to ibis types?
I added an arrow converter (not used, but its in the tests) & a parquet one; certainly can change to just convert parquet types to arrow to ibis.
was organizing like the pandas backend. but yes we could combine all of these into a single file. though this is not user visible anyhow.
changed the title from
WIP: Parquet file interface
ENH: Parquet file interface
Oct 27, 2017
using the example table from https://github.com/apache/arrow/blob/master/python/pyarrow/tests/test_parquet.py#L72
I am not seeing a logical_type for strings in py2; in py3 these are UTF8 (both are BYTE_ARRAY as physical_type). note that the last field is passed in as bytes, so this looks as expected.