Skip to content

[opt](max-compute) avoid repeated location path creation #43355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morningman merged 1 commit into from
Nov 6, 2024
Merged

[opt](max-compute) avoid repeated location path creation #43355

morningman merged 1 commit into from
Nov 6, 2024

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Nov 6, 2024 •
edited by zy-kkk
Loading

What problem does this PR solve?

Related PR: #41803

Problem Summary:
Cherry pick part of code from #41803, some of location path for max compute is constant
and do not need to create every time.
This optimization is already merged in branch-3.0

Check List (For Committer)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No colde files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

  • Release note

    None

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@morningman
Copy link
Contributor Author

morningman commented Nov 6, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TPC-H: Total hot run time: 41692 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aa65ab1a2f5f1508c95ecfcc2b69612d6fcf63a2, data reload: false

------ Round 1 ----------------------------------
q1	17622	7637	7425	7425
q2	2069	185	178	178
q3	10690	1095	1290	1095
q4	10504	876	827	827
q5	7743	3077	3066	3066
q6	238	145	147	145
q7	993	601	598	598
q8	9410	2018	1986	1986
q9	6616	6456	6502	6456
q10	7067	2397	2436	2397
q11	451	285	269	269
q12	402	220	220	220
q13	17752	3037	3034	3034
q14	245	212	218	212
q15	587	515	507	507
q16	627	588	580	580
q17	967	627	584	584
q18	7396	6832	6671	6671
q19	1326	1064	1081	1064
q20	483	185	183	183
q21	3996	3187	3293	3187
q22	1094	1011	1008	1008
Total cold run time: 108278 ms
Total hot run time: 41692 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7349	7361	7383	7361
q2	345	251	254	251
q3	3074	2953	3028	2953
q4	2157	1869	1847	1847
q5	5759	5751	5835	5751
q6	230	139	139	139
q7	2281	1833	1844	1833
q8	3493	3486	3560	3486
q9	8960	8932	8927	8927
q10	3639	3606	3596	3596
q11	607	531	503	503
q12	811	594	654	594
q13	10903	3208	3226	3208
q14	311	286	276	276
q15	589	544	562	544
q16	678	640	697	640
q17	1914	1624	1623	1623
q18	8317	7730	7830	7730
q19	1726	1585	1621	1585
q20	2109	1907	1888	1888
q21	5775	5503	5637	5503
q22	1189	1083	1033	1033
Total cold run time: 72216 ms
Total hot run time: 61271 ms

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

TPC-DS: Total hot run time: 194166 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aa65ab1a2f5f1508c95ecfcc2b69612d6fcf63a2, data reload: false

query1	1289	764	747	747
query2	6232	2130	2130	2130
query3	15775	11505	248	248
query4	33366	23713	23746	23713
query5	3404	456	446	446
query6	264	179	176	176
query7	3978	294	291	291
query8	284	233	229	229
query9	9452	2710	2718	2710
query10	465	248	246	246
query11	18122	15261	15357	15261
query12	161	104	103	103
query13	1577	437	407	407
query14	9712	6734	7098	6734
query15	249	178	193	178
query16	7840	499	464	464
query17	1633	597	598	597
query18	2384	630	618	618
query19	261	195	192	192
query20	120	117	118	117
query21	217	113	112	112
query22	4541	4419	4426	4419
query23	35125	34283	35895	34283
query24	12672	3405	3404	3404
query25	556	414	405	405
query26	1189	179	190	179
query27	2523	289	289	289
query28	7913	2451	2422	2422
query29	685	440	421	421
query30	405	302	309	302
query31	1053	808	804	804
query32	99	56	60	56
query33	782	288	273	273
query34	996	494	518	494
query35	914	789	741	741
query36	1098	951	927	927
query37	198	73	77	73
query38	4472	4206	4288	4206
query39	1488	1440	1425	1425
query40	251	102	102	102
query41	49	49	47	47
query42	109	96	98	96
query43	541	515	505	505
query44	1273	858	819	819
query45	181	166	164	164
query46	1147	692	708	692
query47	1926	1847	1865	1847
query48	432	324	325	324
query49	1081	410	409	409
query50	813	387	388	387
query51	7194	7072	7019	7019
query52	101	98	94	94
query53	265	189	182	182
query54	1220	436	422	422
query55	85	74	82	74
query56	265	259	240	240
query57	1263	1184	1153	1153
query58	211	204	206	204
query59	3321	3160	3285	3160
query60	268	236	247	236
query61	112	110	113	110
query62	896	672	688	672
query63	226	186	184	184
query64	5339	652	642	642
query65	3311	3189	3228	3189
query66	1256	307	298	298
query67	16162	15804	15584	15584
query68	4761	603	579	579
query69	434	254	257	254
query70	1209	1160	1122	1122
query71	378	254	255	254
query72	6027	4073	4027	4027
query73	749	361	353	353
query74	10419	9106	8994	8994
query75	3457	2688	2683	2683
query76	2686	1122	1136	1122
query77	426	277	288	277
query78	10356	9417	9397	9397
query79	1329	608	607	607
query80	949	436	427	427
query81	565	240	239	239
query82	1247	115	119	115
query83	258	170	164	164
query84	242	76	85	76
query85	1419	330	321	321
query86	401	304	312	304
query87	4893	4806	4904	4806
query88	3518	2205	2171	2171
query89	408	295	295	295
query90	1916	193	202	193
query91	145	107	105	105
query92	66	49	51	49
query93	1388	537	547	537
query94	895	296	289	289
query95	345	251	251	251
query96	671	286	280	280
query97	2889	2694	2686	2686
query98	215	207	202	202
query99	1553	1290	1302	1290
Total cold run time: 309006 ms
Total hot run time: 194166 ms

@doris-robot
Copy link

doris-robot commented Nov 6, 2024

ClickBench: Total hot run time: 32.16 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aa65ab1a2f5f1508c95ecfcc2b69612d6fcf63a2, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.06	0.06
query4	1.63	0.11	0.10
query5	0.41	0.41	0.40
query6	1.14	0.65	0.65
query7	0.03	0.02	0.02
query8	0.04	0.03	0.05
query9	0.57	0.50	0.50
query10	0.55	0.55	0.57
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.61	0.60	0.60
query14	2.74	2.83	2.73
query15	0.89	0.82	0.83
query16	0.39	0.37	0.38
query17	0.97	1.04	0.98
query18	0.20	0.19	0.19
query19	1.87	1.78	1.84
query20	0.02	0.00	0.01
query21	15.38	0.62	0.60
query22	3.05	2.42	1.34
query23	16.94	1.10	0.84
query24	3.18	2.23	1.05
query25	0.21	0.24	0.04
query26	0.53	0.13	0.14
query27	0.04	0.05	0.04
query28	9.72	1.11	1.09
query29	12.57	3.25	3.25
query30	0.24	0.06	0.06
query31	2.88	0.38	0.38
query32	3.27	0.46	0.45
query33	2.98	3.04	3.01
query34	17.05	4.49	4.48
query35	4.55	4.56	4.57
query36	0.68	0.51	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 106.42 s
Total hot run time: 32.16 s

zy-kkk
zy-kkk approved these changes Nov 6, 2024
View reviewed changes
Copy link
Member

@zy-kkk zy-kkk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 6, 2024

PR approved by anyone and no changes requested.

@morningman morningman merged commit 52b3660 into apache:master Nov 6, 2024
20 of 21 checks passed
github-actions bot pushed a commit that referenced this pull request Nov 6, 2024
@morningman
[opt](max-compute) avoid repeated location path creation (#43355)
1e98640 
### What problem does this PR solve?

Related PR: #41803

Problem Summary:
Cherry pick part of code from #41803, some of location path for max
compute is constant
and do not need to create every time.
This optimization is already merged in branch-3.0
@github-actions github-actions bot mentioned this pull request Nov 6, 2024
Merged
yiguolei pushed a commit that referenced this pull request Nov 10, 2024
@github-actions @morningman
branch-2.1: [opt](max-compute) avoid repeated location path creation (#…
5ac3aee 
…43383)

Cherry-picked from #43355

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zy-kkk zy-kkk zy-kkk approved these changes

+1 more reviewer

@hubgeter hubgeter hubgeter approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@morningman @doris-robot @zy-kkk @hubgeter @yiguolei