Skip to content

[refine](exchange) Use clone to avoid multiple initializations of the partitioner #47718

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
Mryange wants to merge 1 commit into from
Closed

[refine](exchange) Use clone to avoid multiple initializations of the partitioner #47718

Mryange wants to merge 1 commit into from

Conversation

@Mryange
Copy link
Contributor

@Mryange Mryange commented Feb 10, 2025 •
edited
Loading

What problem does this PR solve?

Similar to PartitionedHashJoinSinkOperatorX. Initialize the partitioner in the Operator and clone it in the local state.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Mryange
Copy link
Contributor Author

Mryange commented Feb 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-H: Total hot run time: 31590 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

------ Round 1 ----------------------------------
q1	17579	5307	5121	5121
q2	2050	309	171	171
q3	10399	1262	757	757
q4	10209	1005	541	541
q5	7530	2398	2326	2326
q6	186	168	133	133
q7	916	739	588	588
q8	9309	1306	1092	1092
q9	4877	4844	4702	4702
q10	6872	2358	1926	1926
q11	489	278	271	271
q12	357	364	228	228
q13	18264	3702	3146	3146
q14	239	226	212	212
q15	521	472	466	466
q16	626	614	586	586
q17	577	870	339	339
q18	6804	6174	6052	6052
q19	1682	964	536	536
q20	318	314	182	182
q21	2900	2102	1913	1913
q22	365	322	302	302
Total cold run time: 103069 ms
Total hot run time: 31590 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5204	5147	5524	5147
q2	228	323	228	228
q3	2153	2697	2315	2315
q4	1418	1799	1336	1336
q5	4261	4151	4152	4151
q6	217	164	125	125
q7	1853	1830	1798	1798
q8	2614	2478	2576	2478
q9	7292	7183	7080	7080
q10	3024	3199	2799	2799
q11	583	519	496	496
q12	687	805	637	637
q13	3428	3935	3279	3279
q14	275	294	267	267
q15	519	474	468	468
q16	670	673	631	631
q17	1173	1622	1334	1334
q18	7560	7281	7327	7281
q19	809	797	832	797
q20	1967	2036	1850	1850
q21	5441	5007	4747	4747
q22	655	602	544	544
Total cold run time: 52031 ms
Total hot run time: 49788 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-DS: Total hot run time: 190405 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

query1	1312	964	944	944
query2	6208	1854	1858	1854
query3	10932	4594	4402	4402
query4	53520	24839	23265	23265
query5	5181	594	487	487
query6	330	233	199	199
query7	4878	503	290	290
query8	333	252	240	240
query9	5485	2489	2499	2489
query10	424	320	252	252
query11	15165	15141	15017	15017
query12	150	107	103	103
query13	1022	519	380	380
query14	11248	6412	6761	6412
query15	200	191	179	179
query16	7066	686	483	483
query17	1084	703	575	575
query18	1532	427	312	312
query19	199	198	157	157
query20	132	120	121	120
query21	204	128	109	109
query22	4555	4501	4385	4385
query23	34126	33409	33190	33190
query24	5822	2455	2437	2437
query25	471	459	396	396
query26	742	273	161	161
query27	1753	503	344	344
query28	2755	2427	2388	2388
query29	571	554	425	425
query30	207	196	158	158
query31	918	892	819	819
query32	74	62	62	62
query33	454	354	304	304
query34	783	859	520	520
query35	813	845	757	757
query36	981	989	918	918
query37	132	106	80	80
query38	4361	4305	4172	4172
query39	1482	1449	1433	1433
query40	211	119	106	106
query41	51	50	52	50
query42	123	111	108	108
query43	503	527	474	474
query44	1331	849	874	849
query45	194	181	168	168
query46	888	1070	666	666
query47	1839	1866	1735	1735
query48	388	418	322	322
query49	720	538	424	424
query50	723	785	437	437
query51	4376	4263	4255	4255
query52	114	103	95	95
query53	252	265	191	191
query54	480	498	402	402
query55	80	79	81	79
query56	274	269	282	269
query57	1169	1167	1110	1110
query58	252	248	252	248
query59	2646	2795	2731	2731
query60	306	286	275	275
query61	119	118	125	118
query62	764	742	687	687
query63	234	203	200	200
query64	1552	1112	766	766
query65	3232	3175	3147	3147
query66	782	394	291	291
query67	15892	15553	15455	15455
query68	4574	790	525	525
query69	518	303	259	259
query70	1202	1054	1133	1054
query71	442	306	266	266
query72	6325	3781	3752	3752
query73	883	763	360	360
query74	9264	9070	9063	9063
query75	3172	3139	2717	2717
query76	3609	1176	735	735
query77	541	364	277	277
query78	10017	10023	9497	9497
query79	2510	819	594	594
query80	585	525	439	439
query81	490	269	231	231
query82	653	159	117	117
query83	183	222	153	153
query84	284	92	72	72
query85	788	338	328	328
query86	358	298	288	288
query87	4420	4540	4350	4350
query88	3568	2227	2222	2222
query89	409	318	278	278
query90	1697	191	189	189
query91	131	148	109	109
query92	71	62	54	54
query93	1932	1011	580	580
query94	717	403	310	310
query95	350	267	253	253
query96	475	564	265	265
query97	2812	2856	2743	2743
query98	234	208	201	201
query99	1340	1398	1248	1248
Total cold run time: 291654 ms
Total hot run time: 190405 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

ClickBench: Total hot run time: 30.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

query1	0.04	0.02	0.04
query2	0.07	0.04	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.41	0.42	0.39
query6	1.16	0.67	0.65
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.60	0.52	0.52
query10	0.59	0.57	0.58
query11	0.15	0.10	0.10
query12	0.15	0.11	0.11
query13	0.62	0.60	0.60
query14	2.73	2.73	2.70
query15	0.92	0.86	0.85
query16	0.37	0.37	0.39
query17	1.02	1.02	1.03
query18	0.21	0.19	0.19
query19	1.94	1.85	2.01
query20	0.01	0.01	0.01
query21	15.35	0.87	0.54
query22	0.74	1.31	0.70
query23	14.81	1.37	0.61
query24	7.38	0.88	0.42
query25	0.50	0.24	0.21
query26	0.67	0.16	0.14
query27	0.05	0.05	0.05
query28	9.34	0.91	0.42
query29	12.54	3.99	3.29
query30	0.25	0.09	0.06
query31	2.81	0.60	0.38
query32	3.22	0.54	0.46
query33	3.10	3.07	3.13
query34	15.76	5.18	4.47
query35	4.54	4.51	4.57
query36	0.65	0.50	0.50
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.03	0.02	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.13 s
Total hot run time: 30.35 s

@Mryange
Copy link
Contributor Author

Mryange commented Feb 10, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-H: Total hot run time: 31855 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

------ Round 1 ----------------------------------
q1	17603	5307	5213	5213
q2	2049	311	165	165
q3	10395	1242	760	760
q4	10215	1016	546	546
q5	7531	2402	2360	2360
q6	194	169	132	132
q7	914	745	604	604
q8	9293	1286	1133	1133
q9	4920	4846	4621	4621
q10	6882	2298	1894	1894
q11	490	272	246	246
q12	361	376	221	221
q13	17766	3772	3113	3113
q14	226	231	206	206
q15	512	472	476	472
q16	609	626	567	567
q17	624	911	365	365
q18	6748	6540	6131	6131
q19	1759	957	577	577
q20	314	339	196	196
q21	2957	2301	2028	2028
q22	365	352	305	305
Total cold run time: 102727 ms
Total hot run time: 31855 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5354	5295	5300	5295
q2	239	325	233	233
q3	2122	2698	2248	2248
q4	1439	1842	1373	1373
q5	4262	4110	4164	4110
q6	212	165	128	128
q7	1867	1860	1771	1771
q8	2645	2704	2631	2631
q9	7179	7122	7090	7090
q10	3018	3206	2818	2818
q11	588	519	489	489
q12	705	762	633	633
q13	3427	4025	3278	3278
q14	284	312	284	284
q15	504	458	469	458
q16	639	666	666	666
q17	1162	1589	1404	1404
q18	7627	7213	7393	7213
q19	856	871	1080	871
q20	1996	2045	1862	1862
q21	5505	4967	4927	4927
q22	626	605	550	550
Total cold run time: 52256 ms
Total hot run time: 50332 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TPC-DS: Total hot run time: 190681 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

query1	1317	989	937	937
query2	6332	1836	1810	1810
query3	11008	4524	4284	4284
query4	54522	25897	23432	23432
query5	4867	614	466	466
query6	313	179	182	179
query7	4869	515	289	289
query8	303	234	217	217
query9	5622	2541	2524	2524
query10	444	308	248	248
query11	15137	15018	15093	15018
query12	150	103	107	103
query13	1034	511	383	383
query14	10460	6878	6902	6878
query15	212	204	199	199
query16	7117	685	504	504
query17	1095	735	611	611
query18	1587	424	316	316
query19	228	185	163	163
query20	126	124	126	124
query21	210	130	104	104
query22	4392	4463	4597	4463
query23	34098	33136	33365	33136
query24	5702	2438	2430	2430
query25	472	468	402	402
query26	695	286	160	160
query27	1664	506	343	343
query28	2776	2444	2432	2432
query29	578	548	429	429
query30	216	190	153	153
query31	906	883	808	808
query32	74	68	56	56
query33	440	370	299	299
query34	783	881	524	524
query35	824	829	744	744
query36	952	1021	914	914
query37	122	107	74	74
query38	4299	4433	4211	4211
query39	1527	1434	1432	1432
query40	207	117	102	102
query41	49	53	50	50
query42	125	113	108	108
query43	501	527	499	499
query44	1351	825	816	816
query45	176	173	175	173
query46	892	1087	680	680
query47	1898	1876	1811	1811
query48	386	415	307	307
query49	686	521	419	419
query50	709	780	412	412
query51	4310	4353	4270	4270
query52	113	125	95	95
query53	247	272	193	193
query54	494	513	432	432
query55	80	83	81	81
query56	282	309	249	249
query57	1173	1183	1155	1155
query58	261	255	257	255
query59	2917	2893	2612	2612
query60	288	281	264	264
query61	146	140	116	116
query62	715	759	693	693
query63	226	194	196	194
query64	1696	1062	821	821
query65	3272	3145	3132	3132
query66	717	417	289	289
query67	15804	15534	15438	15438
query68	3734	800	535	535
query69	485	311	285	285
query70	1192	1120	1120	1120
query71	416	291	279	279
query72	6340	3748	3715	3715
query73	767	770	353	353
query74	8930	9375	8663	8663
query75	3173	3153	2679	2679
query76	2753	1174	759	759
query77	531	353	293	293
query78	10102	10301	9376	9376
query79	972	917	586	586
query80	839	541	460	460
query81	523	271	239	239
query82	431	149	122	122
query83	241	229	156	156
query84	284	92	69	69
query85	752	356	372	356
query86	383	321	271	271
query87	4468	4627	4405	4405
query88	2797	2236	2195	2195
query89	392	311	285	285
query90	1561	196	191	191
query91	136	137	107	107
query92	61	61	57	57
query93	1277	997	590	590
query94	666	389	294	294
query95	363	275	255	255
query96	482	558	276	276
query97	2771	2884	2743	2743
query98	213	210	202	202
query99	1313	1403	1300	1300
Total cold run time: 286542 ms
Total hot run time: 190681 ms

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

ClickBench: Total hot run time: 30.97 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 578793c63ce7de233457f6c7dc7b5fcc8d6937c6, data reload: false

query1	0.03	0.03	0.03
query2	0.08	0.03	0.04
query3	0.24	0.07	0.06
query4	1.62	0.10	0.10
query5	0.40	0.42	0.40
query6	1.16	0.67	0.65
query7	0.02	0.01	0.01
query8	0.04	0.02	0.03
query9	0.60	0.50	0.53
query10	0.57	0.58	0.57
query11	0.15	0.10	0.10
query12	0.15	0.11	0.10
query13	0.62	0.60	0.61
query14	2.70	2.82	2.82
query15	0.92	0.84	0.85
query16	0.39	0.38	0.37
query17	1.05	1.01	1.05
query18	0.22	0.20	0.19
query19	1.95	1.78	2.00
query20	0.01	0.01	0.01
query21	15.38	0.90	0.54
query22	0.74	1.21	0.73
query23	14.83	1.35	0.60
query24	6.92	1.16	1.48
query25	0.49	0.32	0.06
query26	0.50	0.16	0.14
query27	0.05	0.05	0.06
query28	10.21	0.89	0.45
query29	12.87	3.97	3.28
query30	0.25	0.09	0.06
query31	2.84	0.59	0.38
query32	3.24	0.55	0.47
query33	2.99	3.02	2.99
query34	15.78	5.14	4.53
query35	4.54	4.56	4.54
query36	0.65	0.50	0.49
query37	0.09	0.07	0.06
query38	0.05	0.03	0.04
query39	0.03	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.04	0.02
Total cold run time: 105.68 s
Total hot run time: 30.97 s

@doris-robot
Copy link

doris-robot commented Feb 10, 2025

TeamCity be ut coverage result:
Function Coverage: 42.81% (11263/26309)
Line Coverage: 32.79% (94618/288583)
Region Coverage: 31.94% (48500/151854)
Branch Coverage: 27.80% (24454/87952)
Coverage Report: http://coverage.selectdb-in.cc/coverage/578793c63ce7de233457f6c7dc7b5fcc8d6937c6_578793c63ce7de233457f6c7dc7b5fcc8d6937c6/report/index.html

Gabriel39
Gabriel39 requested changes Feb 11, 2025
View reviewed changes
Copy link
Contributor

@Gabriel39 Gabriel39 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This will be done by another PR

@Mryange Mryange marked this pull request as draft February 11, 2025 04:58
@Mryange Mryange closed this Feb 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39 Gabriel39 Gabriel39 requested changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@Mryange @doris-robot @Gabriel39