Skip to content

[refine](pipeline) remove local exchanger type local merge sort#48447

Closed
Mryange wants to merge 2 commits intoapache:masterfrom
Mryange:remove-local-exchange-local-merge-sort
Closed

[refine](pipeline) remove local exchanger type local merge sort#48447
Mryange wants to merge 2 commits intoapache:masterfrom
Mryange:remove-local-exchange-local-merge-sort

Conversation

@Mryange
Copy link
Copy Markdown
Contributor

@Mryange Mryange commented Feb 28, 2025

What problem does this PR solve?

Maintaining a separate local merge sort within LocalExchange can lead to many complex issues. Therefore, it is directly removed, and a separate Operator is implemented.
#48360

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@Mryange
Copy link
Copy Markdown
Contributor Author

Mryange commented Feb 28, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 32080 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 46f96550ef0631905b2e53de3928133b3abd0f41, data reload: false

------ Round 1 ----------------------------------
q1	17578	5328	5198	5198
q2	2056	311	187	187
q3	10481	1248	800	800
q4	10258	1039	554	554
q5	7890	2433	2427	2427
q6	203	172	134	134
q7	916	757	595	595
q8	9316	1321	1159	1159
q9	4924	4680	4927	4680
q10	6875	2315	1901	1901
q11	473	275	252	252
q12	349	356	222	222
q13	17768	3709	3105	3105
q14	222	225	208	208
q15	509	472	445	445
q16	622	623	596	596
q17	621	888	355	355
q18	6588	6240	6264	6240
q19	1520	994	576	576
q20	325	327	183	183
q21	3061	2205	1953	1953
q22	371	342	310	310
Total cold run time: 102926 ms
Total hot run time: 32080 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5285	5239	5223	5223
q2	241	327	234	234
q3	2151	2655	2330	2330
q4	1491	1879	1417	1417
q5	4253	4140	4166	4140
q6	219	168	134	134
q7	1868	2024	1761	1761
q8	2636	2617	2534	2534
q9	7277	7165	7177	7165
q10	3052	3283	2798	2798
q11	570	498	498	498
q12	737	761	593	593
q13	3389	4035	3378	3378
q14	288	303	263	263
q15	511	446	458	446
q16	662	709	645	645
q17	1194	1614	1399	1399
q18	7636	7396	7307	7307
q19	869	906	906	906
q20	1970	2010	1854	1854
q21	5606	5048	4702	4702
q22	638	578	537	537
Total cold run time: 52543 ms
Total hot run time: 50264 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 189717 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 46f96550ef0631905b2e53de3928133b3abd0f41, data reload: false

query1	1320	952	944	944
query2	6186	1922	1944	1922
query3	11008	4358	4402	4358
query4	53390	25504	22944	22944
query5	5198	531	508	508
query6	353	196	180	180
query7	4909	496	287	287
query8	322	242	243	242
query9	5910	2601	2608	2601
query10	437	311	252	252
query11	15279	14998	14939	14939
query12	160	112	105	105
query13	1063	516	395	395
query14	10933	6476	7025	6476
query15	203	199	172	172
query16	6988	655	489	489
query17	1050	699	582	582
query18	1503	420	310	310
query19	206	204	153	153
query20	124	124	120	120
query21	212	123	109	109
query22	4277	4465	4255	4255
query23	33831	33505	33423	33423
query24	5719	2437	2453	2437
query25	459	469	417	417
query26	728	302	173	173
query27	1813	513	324	324
query28	2838	2490	2452	2452
query29	553	558	425	425
query30	211	196	157	157
query31	894	868	802	802
query32	74	63	66	63
query33	453	354	323	323
query34	771	885	521	521
query35	815	830	749	749
query36	944	1002	913	913
query37	125	98	77	77
query38	4163	4170	4128	4128
query39	1492	1474	1475	1474
query40	205	116	111	111
query41	51	50	49	49
query42	122	108	97	97
query43	515	518	481	481
query44	1341	819	804	804
query45	186	180	170	170
query46	890	1066	649	649
query47	1837	1877	1756	1756
query48	382	401	310	310
query49	712	504	415	415
query50	725	763	431	431
query51	4343	4334	4328	4328
query52	114	108	94	94
query53	233	264	195	195
query54	495	503	418	418
query55	83	76	84	76
query56	262	267	247	247
query57	1157	1214	1100	1100
query58	253	246	245	245
query59	2870	2822	2682	2682
query60	294	263	263	263
query61	124	150	117	117
query62	735	755	721	721
query63	237	196	197	196
query64	1857	1140	751	751
query65	3334	3288	3276	3276
query66	797	397	302	302
query67	15850	15479	15071	15071
query68	6526	865	490	490
query69	531	304	269	269
query70	1204	1106	1096	1096
query71	486	290	265	265
query72	5731	3656	3788	3656
query73	1215	739	345	345
query74	8964	9053	8605	8605
query75	3763	3187	2645	2645
query76	4153	1179	741	741
query77	621	383	282	282
query78	10112	9964	9300	9300
query79	2624	809	585	585
query80	684	536	426	426
query81	515	275	240	240
query82	621	121	94	94
query83	175	168	150	150
query84	287	96	74	74
query85	791	345	311	311
query86	391	310	291	291
query87	4456	4405	4363	4363
query88	3566	2213	2191	2191
query89	429	323	291	291
query90	1759	187	188	187
query91	137	140	109	109
query92	78	58	53	53
query93	2240	1044	556	556
query94	671	421	285	285
query95	352	266	258	258
query96	477	576	264	264
query97	3321	3398	3239	3239
query98	237	207	205	205
query99	1324	1387	1295	1295
Total cold run time: 295359 ms
Total hot run time: 189717 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 31.28 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 46f96550ef0631905b2e53de3928133b3abd0f41, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.07	0.07
query4	1.61	0.10	0.10
query5	0.57	0.58	0.55
query6	1.19	0.71	0.72
query7	0.03	0.02	0.02
query8	0.04	0.03	0.04
query9	0.57	0.54	0.51
query10	0.56	0.56	0.57
query11	0.15	0.10	0.11
query12	0.15	0.12	0.12
query13	0.61	0.59	0.59
query14	2.79	2.66	2.83
query15	0.91	0.86	0.85
query16	0.37	0.38	0.37
query17	1.01	1.03	1.03
query18	0.21	0.20	0.19
query19	1.92	1.82	1.97
query20	0.01	0.01	0.02
query21	15.36	0.87	0.54
query22	0.77	1.12	0.67
query23	15.01	1.40	0.68
query24	7.06	1.61	1.35
query25	0.53	0.33	0.09
query26	0.50	0.15	0.14
query27	0.05	0.04	0.05
query28	10.15	0.81	0.42
query29	12.59	3.94	3.30
query30	0.25	0.10	0.06
query31	2.82	0.58	0.38
query32	3.23	0.55	0.48
query33	2.98	2.99	3.04
query34	16.08	5.14	4.52
query35	4.52	4.52	4.48
query36	0.68	0.50	0.48
query37	0.09	0.07	0.06
query38	0.05	0.04	0.04
query39	0.03	0.02	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.02	0.02
Total cold run time: 106.13 s
Total hot run time: 31.28 s

@doris-robot
Copy link
Copy Markdown

BE UT Coverage Report

Increment line coverage 58.33% (7/12) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 44.91% (11965/26643)
Line Coverage 34.42% (100526/292033)
Region Coverage 33.61% (51507/153235)
Branch Coverage 29.39% (26060/88682)

@Mryange
Copy link
Copy Markdown
Contributor Author

Mryange commented Feb 28, 2025

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 31674 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b7874f03ce76e84ad30bd705bdcf5a4d5191a1d6, data reload: false

------ Round 1 ----------------------------------
q1	17576	5213	5057	5057
q2	2049	301	167	167
q3	10406	1253	750	750
q4	10200	1031	533	533
q5	7537	2390	2363	2363
q6	190	166	138	138
q7	889	737	613	613
q8	9304	1326	1108	1108
q9	4978	4756	4884	4756
q10	6828	2330	1865	1865
q11	481	275	246	246
q12	351	358	219	219
q13	17756	3713	3102	3102
q14	227	224	215	215
q15	534	476	474	474
q16	620	636	583	583
q17	591	868	350	350
q18	6867	6163	6137	6137
q19	2566	949	540	540
q20	300	318	180	180
q21	2810	2165	1979	1979
q22	365	341	299	299
Total cold run time: 103425 ms
Total hot run time: 31674 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5245	5118	5159	5118
q2	242	341	232	232
q3	2126	2678	2331	2331
q4	1481	1851	1434	1434
q5	4221	4121	4130	4121
q6	205	166	126	126
q7	1871	1839	1799	1799
q8	2597	2666	2571	2571
q9	7288	7078	7189	7078
q10	3013	3224	2836	2836
q11	577	526	486	486
q12	717	785	657	657
q13	3374	3899	3298	3298
q14	294	318	280	280
q15	496	480	470	470
q16	630	698	652	652
q17	1152	1591	1338	1338
q18	7617	7452	7292	7292
q19	771	862	902	862
q20	1954	2035	1860	1860
q21	5522	5036	4746	4746
q22	627	585	539	539
Total cold run time: 52020 ms
Total hot run time: 50126 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 190979 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b7874f03ce76e84ad30bd705bdcf5a4d5191a1d6, data reload: false

query1	1285	948	934	934
query2	6193	1850	1842	1842
query3	11155	4670	4629	4629
query4	25746	23510	23445	23445
query5	4761	686	501	501
query6	316	244	188	188
query7	3983	488	302	302
query8	298	234	219	219
query9	8483	2580	2608	2580
query10	435	298	250	250
query11	15237	15152	14898	14898
query12	162	113	105	105
query13	1582	520	387	387
query14	8873	6492	6072	6072
query15	212	189	177	177
query16	7342	655	472	472
query17	1174	744	640	640
query18	2006	435	334	334
query19	202	201	171	171
query20	141	127	124	124
query21	207	144	109	109
query22	4496	4811	4436	4436
query23	34389	33413	33441	33413
query24	7897	2447	2423	2423
query25	512	459	393	393
query26	1280	283	150	150
query27	2562	484	328	328
query28	4575	2510	2420	2420
query29	718	558	437	437
query30	231	205	156	156
query31	925	891	818	818
query32	75	68	68	68
query33	523	362	301	301
query34	805	883	525	525
query35	797	847	761	761
query36	996	1018	926	926
query37	121	104	74	74
query38	4093	4236	4257	4236
query39	1473	1439	1477	1439
query40	203	117	103	103
query41	50	54	49	49
query42	119	104	102	102
query43	499	533	481	481
query44	1308	805	802	802
query45	185	180	163	163
query46	878	1078	630	630
query47	1810	1893	1864	1864
query48	383	416	311	311
query49	764	515	417	417
query50	734	760	412	412
query51	4271	4289	4246	4246
query52	114	114	117	114
query53	230	271	201	201
query54	491	493	438	438
query55	86	92	82	82
query56	268	295	258	258
query57	1148	1133	1090	1090
query58	250	252	234	234
query59	2761	2975	2653	2653
query60	287	285	275	275
query61	120	115	149	115
query62	791	711	674	674
query63	233	193	196	193
query64	4068	1077	665	665
query65	3257	3205	3211	3205
query66	1228	399	315	315
query67	16221	15495	15391	15391
query68	8470	869	499	499
query69	531	289	272	272
query70	1215	1136	1084	1084
query71	466	302	270	270
query72	5582	3501	3485	3485
query73	797	727	345	345
query74	9162	9192	9015	9015
query75	3709	3179	2717	2717
query76	3697	1196	729	729
query77	774	355	272	272
query78	9948	10314	9286	9286
query79	2330	811	591	591
query80	591	511	462	462
query81	496	277	248	248
query82	611	123	91	91
query83	177	173	145	145
query84	241	85	76	76
query85	753	355	393	355
query86	327	310	276	276
query87	4449	4549	4631	4549
query88	3397	2214	2200	2200
query89	410	319	288	288
query90	1858	193	201	193
query91	132	135	117	117
query92	70	57	66	57
query93	1316	1042	572	572
query94	662	403	300	300
query95	348	264	343	264
query96	472	564	276	276
query97	3257	3377	3258	3258
query98	234	212	204	204
query99	1431	1394	1317	1317
Total cold run time: 276931 ms
Total hot run time: 190979 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b7874f03ce76e84ad30bd705bdcf5a4d5191a1d6, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.04	0.03
query3	0.23	0.06	0.07
query4	1.61	0.10	0.10
query5	0.56	0.56	0.57
query6	1.20	0.72	0.71
query7	0.03	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.53	0.51
query10	0.57	0.56	0.56
query11	0.16	0.11	0.10
query12	0.14	0.11	0.12
query13	0.61	0.60	0.60
query14	2.66	2.80	2.67
query15	0.92	0.84	0.84
query16	0.39	0.37	0.37
query17	1.03	1.02	1.01
query18	0.21	0.20	0.19
query19	1.86	1.77	2.02
query20	0.01	0.01	0.01
query21	15.37	0.88	0.53
query22	0.74	1.23	0.64
query23	14.92	1.37	0.61
query24	7.08	1.34	0.89
query25	0.52	0.26	0.08
query26	0.50	0.16	0.14
query27	0.05	0.05	0.05
query28	10.26	0.85	0.43
query29	12.60	3.98	3.33
query30	0.25	0.09	0.06
query31	2.81	0.60	0.39
query32	3.22	0.55	0.47
query33	3.00	3.00	3.08
query34	15.86	5.13	4.52
query35	4.53	4.56	4.47
query36	0.65	0.50	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.79 s
Total hot run time: 30.69 s

@hello-stephen
Copy link
Copy Markdown
Contributor

BE UT Coverage Report

Increment line coverage 53.85% (7/13) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 44.91% (11973/26659)
Line Coverage 34.42% (100589/292278)
Region Coverage 33.62% (51553/153360)
Branch Coverage 29.37% (26066/88750)

@Mryange Mryange marked this pull request as draft March 1, 2025 11:04
@Mryange Mryange closed this Mar 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants