[Python][Parquet] Use pypi condition package to filter partitions in a user friendly way

[asfimport]

I created the pypi condition package to allow user friendly expression of conditions. For example, a condition can be written as:

(f.A <= 3 or f.B != 'b1') and f.C == ['c1', 'c2'] 

where A, B, C are partition keys.

For usage details, please see its document at: 

https://condition.readthedocs.io/en/latest/usage.html

 

Arbitrary condition objects can be converted to pyarrow's filter by calling its

to_pyarrow_filter() method:

https://condition.readthedocs.io/en/latest/usage.html#pyarrow-partition-filtering

The above method will normalize the condition to conform to pyarrow filter specification.

 

Furthermore, the condition object be directly used to evaluate partition paths. This can replace the current complex filtering codes. (both native and python)

For max efficiency, filtering with the condition object can be done in the below ways:

1. read the paths in chunks to keep the memory footprint small;

2. parse the paths to be a pandas dataframe;

3. use condition.query(dataframe) to get the filtered dataframe of path.

4. use numexpr backend for dataframe query for efficiency.

   Please discuss.

Reporter: Weiyang Zhao / @wyzhao

_{Note: This issue was originally created as ARROW-11566. Please see the migration documentation for further details.}

[asfimport]

Todd Farmer / @toddfarmer:
This issue was last updated over 90 days ago, which may be an indication it is no longer being actively worked. To better reflect the current state, the issue is being unassigned. Please feel free to re-take assignment of the issue if it is being actively worked, or if you plan to start that work soon.

[github-actions]

This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.