semi/anti join

[marsupialtail]

Problem description

Currently if you want to do BOTH a semi join and an anti join, you have to, well, do them both. Most of the work is duplicated.

It would be amazing if Polars support something like a ""semi/anti" join where it produces two results, one for the semi join and one for the anti join in one pass.

[MarcoGorelli]

thanks @marsupialtail for the request

I think @orlp had a suggestion to do materialise multiple expressions together

I think this is a good idea, because otherwise, splitting lazyframes means you risk "shooting yourself in the foot" with repeated computations

For example

df_collected = df.collect()
other_collected = other.collect()
semi, anti = (
    df_collected.join(other_collected, on='acoustic_data', how='semi'),
    df_collected.join(other_collected, on='acoustic_data', how='anti')
)

is twice as fast as

%%time
semi, anti = pl.collect_all(
    [
        df.join(other, on='acoustic_data', how='semi'),
        df.join(other, on='acoustic_data', how='anti')
    ]
)

notebook where I time this: https://www.kaggle.com/code/marcogorelli/semi-anti-join-in-polars?scriptVersionId=155140639

[orlp]

In the future we'll support smart join sharing like this under the hood using the optimizing engine.