[C++][Python] Expose sorting_columns in RowGroupMetaData for Parquet files

[ei-grad]

Summary

Currently, the pyarrow.parquet.RowGroupMetaData class does not expose the sorting_columns information available in the Parquet format's RowGroup struct. This information is useful for users who need to understand the local sorting order of columns within each RowGroup. It would be beneficial to expose this information in the RowGroupMetaData class.

Details

The Parquet format includes an optional sorting_columns field in the RowGroup struct, which stores information about the sorting order of columns within the RowGroup. This information is defined in the SortingColumn struct in the parquet.thrift file:

struct SortingColumn {
  1: required int32 column_idx;
  2: required bool descending;
  3: optional bool nulls_first;
}

In the RowGroup struct, the sorting_columns field is defined as follows:

struct RowGroup {
  1: required list<ColumnChunk> columns;
  2: required i64 total_byte_size;
  3: required i64 num_rows;
  4: optional list<SortingColumn> sorting_columns;
}

However, the pyarrow.parquet.RowGroupMetaData class does not expose this information. As a result, users cannot access the local sorting information of columns within RowGroups.

Proposal

I propose adding a new method or property in the RowGroupMetaData class to expose the sorting_columns information. This could be implemented as a new method, such as get_sorting_columns(), or as a property, such as sorting_columns. The output should include the column index, sorting order (ascending or descending), and whether null values appear first or last in the sorted order.

Use Case

Users working with sorted Parquet files can benefit from understanding the local sorting order of columns within RowGroups. This information is particularly useful when analyzing large datasets or performing operations that require knowledge of the sort order, such as range queries, filtering, or merging.

By exposing the sorting_columns information in the RowGroupMetaData class, users can more easily work with sorted Parquet files and perform advanced data processing operations.

Component(s)

Python

[jorisvandenbossche]

@ei-grad thanks for opening the issue! I think that would be a welcome enhancement.

Note that this is also not yet exposed in the C++ parquet::RowGroupMetaData (which wraps the raw parquet::format::RowGroup, which is the auto-generated equivalent of the thrift struct you mentioned). So a first step would be to expose it in C++, and then the Python bindings can expose it as well.

[mapleFU]

I can help with the C++ part. Will finish it tonight

[mapleFU]

Draft an issue, will test it later: https://github.com/apache/arrow/pull/35351/files

[mapleFU]

@wjones127 Hi will, I only solve the C++ part. Should we reuse this issue, or I should create another issue for C++/Python, and use that issue?

[wjones127]

Ah sorry I didn't notice that. I think we generally want one-to-one correspondence with issues and pull requests, so i should have created a separate sub-issue for the C++ part. I think its fine for now if we re-use the issue, right @jorisvandenbossche ?

[jorisvandenbossche]

Yes, let's just re-use it

[mapleFU]

I'm not familiar with Python part, would you mind do this, or tell me the code I can take use for reference?

[wjones127]

I can work on that soon.

[mapleFU]

Please not assign to me =_=

[mapleFU]

@judahrand would you mind reply here? we can only assign to the one replied here

[judahrand]

Sure!