perf(rust,python): avoid quadratic exclude behaviour when selecting against dtypes and/or wildcards

[alexander-beedie]

Closes #8947.

Huge speedup when excluding on wide frames; multiple orders of magnitude as the number of cols gets large. (Even though it's not noticeable on smaller frames, the quadratic behaviour really catches up with you as you head into the tens of thousands of columns 😅).

Changed prepare_excluded to return a HashSet as the number of items it can hold may be large, so we want the O(1) lookup; the dtypes param doesn't need the same treatment as even in the most extreme case there can only be a handful of them. Also moved the exclusion-match (inside expand_dtypes) up into the iter_fields filter.

Example

from codetiming import Timer
import polars as pl

for n in ( 
    # ↓ fast enough not to notice
    100,
    500,
    1_000,
    2_500,
    5_000,
    7_500,
    10_000,
    # ↓ "highway to the danger zone"
    50_000,
    75_000,
    100_000,
    175_000,
    250_000,
    500_000,
    1_000_000,
):
    df = pl.DataFrame( schema={f"c{i}":pl.Int32 for i in range(n)} )
    with Timer():
        _ = df.select( pl.exclude(pl.NUMERIC_DTYPES) )

Results

Timings:

Hardware: Apple Silicon M2 Max Pro
Compiled with: maturin develop --release -- -C target-cpu=native

n cols	this pr	main	speedup
100	0.0001	0.0001	1x
500	0.0002	0.0006	3x
1,000	0.0004	0.0022	5x
2,500	0.0008	0.0089	11x
5,000	0.0015	0.0423	28x
7,500	0.0018	0.0845	46x
10,000	0.0022	0.1555	70x
50,000	0.0113	3.3557	296x
75,000	0.0187	8.2587	441x
100,000	0.0248	14.9413	602x
175,000	0.0418	30.139	721x
250,000	0.0688	65.9686	958x
500,000	0.1478	333.4362	2255x
1,000,000	0.3226	1527.7112	4735x

Plot:

Orange: current main.
Blue: this pr.

[mcrumiller]

What library did you use for the figure?

[alexander-beedie]

What library did you use for the figure?

An elderly copy of Excel 🤣

[ritchie46]

Good catch!