feat(python): extend filter capabilities with new support for *args predicates, **kwargs constraints, and chained boolean masks

[alexander-beedie]

Closes #11642, supersedes #11674.

Ended up going a little deeper down the rabbit hole so moved it into my own branch to finish off as I was having issues rebasing / force-pushing to a branch I don't own (some sort of credentials/token issue? 🤷); kudos to @mcrumiller for getting things started with the initial implementation!

Miscellaneous

• Turns out we can cleanly disambiguate new/old usage of the predicate keyword, so only warns on old-style usage.
• Noticed a fast-path for recognising numpy-based boolean masks (and int sequences), so integrated this too.

New Features

Adds support for:

• *predicates (one or more predicates as positional arguments).
• **constraints (col == value matches as keyword arguments).
• resolving chains of boolean masks (previously only allowed one).

Examples

More than one predicate, now we have a positional args option.
Implicitly combines predicates with all_horizontal:

df.filter( 
    pl.col("a") >= 0, 
    pl.col("b").is_not_null(),
    pl.col("c").str.starts_with("!"),
 )

Splat predicates assembled elsewhere straight into filter without having to combine them yourself:

df.filter( *conditions )

Constrain rows using **kwargs format (essentially the same API that we offer in with_columns):

df.filter( 
    x = 0,
    y = "xxx",
    z = date.today(),
)

Mix and match the two formats freely:

df.filter( *conditions, value=0, start_date=date.today() )

Easily filter against a Pydantic/FastAPI Model object:

from pydantic import BaseModel
from datetime import date
import polars as pl

class DataModel(BaseModel):
    x: date
    y: str
    z: int
    
model_data = [
    DataModel( x=date.today(), y="aa", z=123 ),
    DataModel( x=date.today(), y="bb", z=456 ),
    DataModel( x=date.today(), y="cc", z=789 ),
]

df = pl.DataFrame( data=model_data )
# shape: (3, 3)
# ┌────────────┬─────┬─────┐
# │ x          ┆ y   ┆ z   │
# │ ---        ┆ --- ┆ --- │
# │ date       ┆ str ┆ i64 │
# ╞════════════╪═════╪═════╡
# │ 2023-10-14 ┆ aa  ┆ 123 │
# │ 2023-10-14 ┆ bb  ┆ 456 │
# │ 2023-10-14 ┆ cc  ┆ 789 │
# └────────────┴─────┴─────┘

df.filter( **model_data[1].model_dump() )
# shape: (1, 3)
# ┌────────────┬─────┬─────┐
# │ x          ┆ y   ┆ z   │
# │ ---        ┆ --- ┆ --- │
# │ date       ┆ str ┆ i64 │
# ╞════════════╪═════╪═════╡
# │ 2023-10-14 ┆ bb  ┆ 456 │
# └────────────┴─────┴─────┘

Chain boolean masks (previously only allowed one per filter invocation):

mask1 = [True, False, True]
mask2 = [False, True, True]

df.filter( mask1, mask2 )
# shape: (1, 3)
# ┌────────────┬─────┬─────┐
# │ x          ┆ y   ┆ z   │
# │ ---        ┆ --- ┆ --- │
# │ date       ┆ str ┆ i64 │
# ╞════════════╪═════╪═════╡
# │ 2023-10-14 ┆ cc  ┆ 789 │
# └────────────┴─────┴─────┘

[stinodego]

I think this feature will be a nice quality-of-life upgrade to an often-used method!

I wonder if we can't just use parse_as_list_of_expressions here? There seems to be a lot of duplicated parsing logic here.

[alexander-beedie]

I wonder if we can't just use parse_as_list_of_expressions here?

We can't hand off to that completely, as the boolean masks need to be identified separately (update: it can be used to unify parsing of a single-list passed to *predicates without splatting though, so I smoothed that out👌).

[stinodego]

Great stuff, Alex!