sorted_merge_join

[ritchie46]

Sorted merge join optimization.

closes #3474

Tested on:

os.environ["POLARS_VERBOSE"] = "1"
for i in range(0, 300):
    np.random.seed(i)
    n = 10_000
    df_a = pl.DataFrame({
        'a': np.sort(np.random.randint(0, n // 2, n))
    }).with_row_count("row_a")


    df_b = pl.DataFrame({
        'a': np.sort(np.random.randint(0, n // 2, n // 2))
    }).with_row_count("row_b")

    for how in ["left", "inner"]:
        
        # hash join
        out_hash_join = df_a.join(df_b, on="a", how=how)

        # sorted merge join
        out_sorted_merge_join = df_a.with_column(
            pl.col("a").set_sorted()
        ).join(df_b.with_column(
            pl.col("a").set_sorted()
        ), on="a", how=how)

        assert out_hash_join.frame_equal(out_sorted_merge_join)

Performance

[codecov-commenter]

Codecov Report

Merging #3505 (5d0babc) into master (e5f3173) will decrease coverage by 0.27%.
The diff coverage is 3.17%.

@@            Coverage Diff             @@
##           master    #3505      +/-   ##
==========================================
- Coverage   61.56%   61.29%   -0.28%     
==========================================
  Files         424      427       +3     
  Lines       71057    71429     +372     
==========================================
+ Hits        43747    43782      +35     
- Misses      27310    27647     +337     

Impacted Files	Coverage Δ
polars/polars-arrow/src/data_types.rs	60.00% <ø> (ø)
polars/polars-arrow/src/kernels/mod.rs	63.69% <ø> (ø)
...lars/polars-arrow/src/kernels/sorted_join/inner.rs	0.00% <0.00%> (ø)
...olars/polars-arrow/src/kernels/sorted_join/left.rs	0.00% <0.00%> (ø)
polars/polars-arrow/src/kernels/take.rs	68.14% <ø> (-0.25%)	⬇️
polars/polars-arrow/src/lib.rs	0.00% <ø> (ø)
...lars/polars-core/src/frame/hash_join/sort_merge.rs	7.44% <7.44%> (ø)
polars/polars-core/src/frame/hash_join/mod.rs	41.51% <28.57%> (-0.03%)	⬇️
py-polars/polars/io.py	68.51% <0.00%> (-6.97%)	⬇️
polars/polars-core/src/series/ops/round.rs	41.89% <0.00%> (-6.76%)	⬇️
... and 20 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update e5f3173...5d0babc. Read the comment docs.