Skip to content

remove repartition exec from coalesce batches optimizer #19239

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
jizezhang wants to merge 2 commits into
base: main
Choose a base branch
from
Open

remove repartition exec from coalesce batches optimizer #19239

jizezhang wants to merge 2 commits into from
+977 −1,291

Conversation

@jizezhang
Copy link
Contributor

@jizezhang jizezhang commented Dec 9, 2025 •
edited
Loading

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Removes RepartitionExec from CoalesceBatches optimizer.

Are these changes tested?

Yes

Are there any user-facing changes?

No

@github-actions github-actions bot added optimizer Optimizer rules core Core DataFusion crate sqllogictest SQL Logic Tests (.slt) labels Dec 9, 2025
@jizezhang jizezhang marked this pull request as ready for review December 10, 2025 06:24
@jizezhang jizezhang mentioned this pull request Dec 10, 2025
Open
@alamb alamb mentioned this pull request Dec 11, 2025
Open
36 tasks
@alamb
Copy link
Contributor

alamb commented Dec 11, 2025

run benchmarks

@alamb
Copy link
Contributor

alamb commented Dec 11, 2025

run benchmark tpch_mem10

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 11, 2025

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf) to 20870c1 diff using: tpch_mem clickbench_partitioned clickbench_extended
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 12, 2025

🤖: Benchmark completed

Details 

Comparing HEAD and jizezhang_main
--------------------
Benchmark clickbench_extended.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ jizezhang_main ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 0     │  2733.19 ms │     2720.03 ms │ no change │
│ QQuery 1     │  1281.78 ms │     1252.52 ms │ no change │
│ QQuery 2     │  2377.84 ms │     2424.84 ms │ no change │
│ QQuery 3     │  1160.92 ms │     1168.87 ms │ no change │
│ QQuery 4     │  2324.43 ms │     2290.43 ms │ no change │
│ QQuery 5     │ 28592.61 ms │    28760.63 ms │ no change │
│ QQuery 6     │  4037.19 ms │     3970.23 ms │ no change │
│ QQuery 7     │  3695.67 ms │     3737.12 ms │ no change │
└──────────────┴─────────────┴────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary             ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)             │ 46203.63ms │
│ Total Time (jizezhang_main)   │ 46324.67ms │
│ Average Time (HEAD)           │  5775.45ms │
│ Average Time (jizezhang_main) │  5790.58ms │
│ Queries Faster                │          0 │
│ Queries Slower                │          0 │
│ Queries with No Change        │          8 │
│ Queries with Failure          │          0 │
└───────────────────────────────┴────────────┘
--------------------
Benchmark clickbench_partitioned.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━┓
┃ Query        ┃        HEAD ┃ jizezhang_main ┃        Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━┩
│ QQuery 0     │     2.57 ms │        2.25 ms │ +1.14x faster │
│ QQuery 1     │    52.05 ms │       52.40 ms │     no change │
│ QQuery 2     │   134.07 ms │      134.10 ms │     no change │
│ QQuery 3     │   157.59 ms │      154.23 ms │     no change │
│ QQuery 4     │  1177.17 ms │     1092.07 ms │ +1.08x faster │
│ QQuery 5     │  1555.61 ms │     1502.54 ms │     no change │
│ QQuery 6     │     2.14 ms │        2.19 ms │     no change │
│ QQuery 7     │    56.24 ms │       57.00 ms │     no change │
│ QQuery 8     │  1508.89 ms │     1430.78 ms │ +1.05x faster │
│ QQuery 9     │  1970.06 ms │     1881.53 ms │     no change │
│ QQuery 10    │   371.09 ms │      374.63 ms │     no change │
│ QQuery 11    │   416.93 ms │      423.24 ms │     no change │
│ QQuery 12    │  1440.40 ms │     1365.87 ms │ +1.05x faster │
│ QQuery 13    │  2049.91 ms │     2021.63 ms │     no change │
│ QQuery 14    │  1300.92 ms │     1248.05 ms │     no change │
│ QQuery 15    │  1307.90 ms │     1243.04 ms │     no change │
│ QQuery 16    │  2665.57 ms │     2700.25 ms │     no change │
│ QQuery 17    │  2677.16 ms │     2689.07 ms │     no change │
│ QQuery 18    │  5058.27 ms │     5181.21 ms │     no change │
│ QQuery 19    │   126.21 ms │      124.54 ms │     no change │
│ QQuery 20    │  1896.68 ms │     1911.49 ms │     no change │
│ QQuery 21    │  2174.96 ms │     2189.22 ms │     no change │
│ QQuery 22    │  3676.22 ms │     3730.96 ms │     no change │
│ QQuery 23    │ 12348.34 ms │    12302.89 ms │     no change │
│ QQuery 24    │   208.97 ms │      211.21 ms │     no change │
│ QQuery 25    │   464.76 ms │      471.47 ms │     no change │
│ QQuery 26    │   220.82 ms │      208.49 ms │ +1.06x faster │
│ QQuery 27    │  2708.31 ms │     2710.37 ms │     no change │
│ QQuery 28    │ 23987.85 ms │    24389.78 ms │     no change │
│ QQuery 29    │   974.58 ms │      940.27 ms │     no change │
│ QQuery 30    │  1352.37 ms │     1309.12 ms │     no change │
│ QQuery 31    │  1365.80 ms │     1327.44 ms │     no change │
│ QQuery 32    │  5194.56 ms │     4840.81 ms │ +1.07x faster │
│ QQuery 33    │  5948.34 ms │     5925.71 ms │     no change │
│ QQuery 34    │  6060.74 ms │     6119.94 ms │     no change │
│ QQuery 35    │  2005.86 ms │     1924.80 ms │     no change │
│ QQuery 36    │   118.66 ms │      117.42 ms │     no change │
│ QQuery 37    │    54.36 ms │       55.94 ms │     no change │
│ QQuery 38    │   117.99 ms │      116.94 ms │     no change │
│ QQuery 39    │   188.85 ms │      196.87 ms │     no change │
│ QQuery 40    │    46.58 ms │       43.80 ms │ +1.06x faster │
│ QQuery 41    │    40.04 ms │       41.09 ms │     no change │
│ QQuery 42    │    34.13 ms │       33.88 ms │     no change │
└──────────────┴─────────────┴────────────────┴───────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Benchmark Summary             ┃            ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ Total Time (HEAD)             │ 95220.52ms │
│ Total Time (jizezhang_main)   │ 94800.51ms │
│ Average Time (HEAD)           │  2214.43ms │
│ Average Time (jizezhang_main) │  2204.66ms │
│ Queries Faster                │          7 │
│ Queries Slower                │          0 │
│ Queries with No Change        │         36 │
│ Queries with Failure          │          0 │
└───────────────────────────────┴────────────┘
--------------------
Benchmark tpch_mem_sf1.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Query        ┃      HEAD ┃ jizezhang_main ┃    Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ QQuery 1     │ 129.27 ms │      132.98 ms │ no change │
│ QQuery 2     │  27.66 ms │       27.10 ms │ no change │
│ QQuery 3     │  37.19 ms │       37.92 ms │ no change │
│ QQuery 4     │  28.76 ms │       28.28 ms │ no change │
│ QQuery 5     │  87.45 ms │       86.95 ms │ no change │
│ QQuery 6     │  19.85 ms │       19.87 ms │ no change │
│ QQuery 7     │ 227.31 ms │      227.85 ms │ no change │
│ QQuery 8     │  36.11 ms │       35.44 ms │ no change │
│ QQuery 9     │ 106.23 ms │      101.00 ms │ no change │
│ QQuery 10    │  63.19 ms │       63.64 ms │ no change │
│ QQuery 11    │  17.70 ms │       18.54 ms │ no change │
│ QQuery 12    │  52.76 ms │       52.15 ms │ no change │
│ QQuery 13    │  47.73 ms │       48.17 ms │ no change │
│ QQuery 14    │  13.77 ms │       14.03 ms │ no change │
│ QQuery 15    │  24.71 ms │       24.06 ms │ no change │
│ QQuery 16    │  25.16 ms │       24.86 ms │ no change │
│ QQuery 17    │ 150.79 ms │      150.25 ms │ no change │
│ QQuery 18    │ 279.74 ms │      278.43 ms │ no change │
│ QQuery 19    │  37.56 ms │       37.52 ms │ no change │
│ QQuery 20    │  49.26 ms │       48.52 ms │ no change │
│ QQuery 21    │ 313.83 ms │      322.16 ms │ no change │
│ QQuery 22    │  17.49 ms │       17.33 ms │ no change │
└──────────────┴───────────┴────────────────┴───────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━┓
┃ Benchmark Summary             ┃           ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━┩
│ Total Time (HEAD)             │ 1793.52ms │
│ Total Time (jizezhang_main)   │ 1797.05ms │
│ Average Time (HEAD)           │   81.52ms │
│ Average Time (jizezhang_main) │   81.68ms │
│ Queries Faster                │         0 │
│ Queries Slower                │         0 │
│ Queries with No Change        │        22 │
│ Queries with Failure          │         0 │
└───────────────────────────────┴───────────┘

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 12, 2025

🤖 ./gh_compare_branch.sh gh_compare_branch.sh Running
Linux aal-dev 6.14.0-1018-gcp #19~24.04.1-Ubuntu SMP Wed Sep 24 23:23:09 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Comparing jizezhang/main (446f6cf) to 20870c1 diff using: tpch_mem10
Results will be posted here when complete

@alamb-ghbot
Copy link

alamb-ghbot commented Dec 12, 2025

🤖: Benchmark completed

Details 

Comparing HEAD and jizezhang_main
--------------------
Benchmark tpch_mem_sf10.json
--------------------
┏━━━━━━━━━━━━━━┳━━━━━━┳━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━┓
┃ Query        ┃ HEAD ┃ jizezhang_main ┃       Change ┃
┡━━━━━━━━━━━━━━╇━━━━━━╇━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━┩
│ QQuery 1     │ FAIL │           FAIL │ incomparable │
│ QQuery 2     │ FAIL │           FAIL │ incomparable │
│ QQuery 3     │ FAIL │           FAIL │ incomparable │
│ QQuery 4     │ FAIL │           FAIL │ incomparable │
│ QQuery 5     │ FAIL │           FAIL │ incomparable │
│ QQuery 6     │ FAIL │           FAIL │ incomparable │
│ QQuery 7     │ FAIL │           FAIL │ incomparable │
│ QQuery 8     │ FAIL │           FAIL │ incomparable │
│ QQuery 9     │ FAIL │           FAIL │ incomparable │
│ QQuery 10    │ FAIL │           FAIL │ incomparable │
│ QQuery 11    │ FAIL │           FAIL │ incomparable │
│ QQuery 12    │ FAIL │           FAIL │ incomparable │
│ QQuery 13    │ FAIL │           FAIL │ incomparable │
│ QQuery 14    │ FAIL │           FAIL │ incomparable │
│ QQuery 15    │ FAIL │           FAIL │ incomparable │
│ QQuery 16    │ FAIL │           FAIL │ incomparable │
│ QQuery 17    │ FAIL │           FAIL │ incomparable │
│ QQuery 18    │ FAIL │           FAIL │ incomparable │
│ QQuery 19    │ FAIL │           FAIL │ incomparable │
│ QQuery 20    │ FAIL │           FAIL │ incomparable │
│ QQuery 21    │ FAIL │           FAIL │ incomparable │
│ QQuery 22    │ FAIL │           FAIL │ incomparable │
└──────────────┴──────┴────────────────┴──────────────┘
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┓
┃ Benchmark Summary             ┃        ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━┩
│ Total Time (HEAD)             │ 0.00ms │
│ Total Time (jizezhang_main)   │ 0.00ms │
│ Average Time (HEAD)           │ 0.00ms │
│ Average Time (jizezhang_main) │ 0.00ms │
│ Queries Faster                │      0 │
│ Queries Slower                │      0 │
│ Queries with No Change        │      0 │
│ Queries with Failure          │     22 │
└───────────────────────────────┴────────┘

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

core Core DataFusion crate optimizer Optimizer rules sqllogictest SQL Logic Tests (.slt)

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Integrate BatchCoalescer into RepartitionExec and remove from CoalesceBatches optimization rule

3 participants

@jizezhang @alamb @alamb-ghbot