[C++][Python] Support AWS Firehose partition_scheme implementation for Parquet datasets

[asfimport]

I'd like to be able to read a ParquetDataset generated by AWS Firehose.

The only implementation at the time of writting was the partition scheme created by hive (year=2018/month=01/day=11).

AWS Firehose partition scheme is a little bit different (2018/01/11).

 

Thanks

Reporter: Pablo Javier Takara

Related issues:

• [Python][Dataset] Support using dataset API in pyarrow.parquet with a minimal ParquetDataset shim (depends upon)

_{Note: This issue was originally created as ARROW-2882. Please see the migration documentation for further details.}

[asfimport]

Wes McKinney / @wesm:
Thanks [~IceS2] I updated the title and added a component and label

[asfimport]

Wes McKinney / @wesm:
I added the C++ component since this will be handled as part of the Datasets project

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
This kind of partitioning scheme is already implemented in the Datasets project, and exposed in the python bindings.

You can do:

import pyarrow.dataset as ds

ds.dataset("root_directory/", partitioning=ds.partitioning(["year", "month", "day"])

to give the names for the parts of the file path. Alternatively, you can also pass an actual schema, in which case you specify data types per field as well, instead of letting it be inferred from the file path.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
With ARROW-8039, this is now also exposed in the existing pq.ParquetDataset, when using use_legacy_dataset=False.

[asfimport]

Joris Van den Bossche / @jorisvandenbossche:
Let's declare this as resolved, now the new Datasets API supports this.