Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improvement](mtmv) improve mv rewrite performance by reuse the shuttled expression #37197

Merged
merged 2 commits into from
Jul 5, 2024
Merged

[improvement](mtmv) improve mv rewrite performance by reuse the shuttled expression #37197

merged 2 commits into from
Jul 5, 2024

Conversation

seawinde
Copy link
Contributor

@seawinde seawinde commented Jul 3, 2024

Proposed changes

Optimizations:

  1. Expression shuttle is expensive in materialized view rewritting, So reuse the shuttled expression.
  2. Generate shuttledExpressions by planOutput is also expensive, so generate and store in struct info once and used later

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@seawinde
Copy link
Contributor Author

seawinde commented Jul 3, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39639 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f367e1c12510f72ed2667ae3b1b1a31229e90a24, data reload: false

------ Round 1 ----------------------------------
q1	17617	4403	4268	4268
q2	2008	193	205	193
q3	10458	1129	1101	1101
q4	10194	750	856	750
q5	7570	2639	2594	2594
q6	216	138	140	138
q7	942	596	600	596
q8	9236	2066	2025	2025
q9	8748	6500	6461	6461
q10	8994	3716	3727	3716
q11	470	240	243	240
q12	400	230	238	230
q13	18844	2980	2997	2980
q14	273	220	234	220
q15	540	477	484	477
q16	525	380	384	380
q17	965	692	596	596
q18	8043	7504	7333	7333
q19	7456	1485	1481	1481
q20	668	337	343	337
q21	4818	3184	3202	3184
q22	411	344	339	339
Total cold run time: 119396 ms
Total hot run time: 39639 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4378	4227	4246	4227
q2	379	278	253	253
q3	3002	2993	2904	2904
q4	2005	1657	1759	1657
q5	5667	5530	5459	5459
q6	221	141	137	137
q7	2212	1844	1795	1795
q8	3238	3389	3450	3389
q9	8704	8727	8773	8727
q10	4128	3876	3735	3735
q11	601	494	496	494
q12	816	639	652	639
q13	16311	3203	3202	3202
q14	321	293	280	280
q15	526	486	487	486
q16	499	433	454	433
q17	1862	1525	1504	1504
q18	8147	7886	7930	7886
q19	1754	1653	1672	1653
q20	2184	1889	1853	1853
q21	5227	5042	4732	4732
q22	628	569	585	569
Total cold run time: 72810 ms
Total hot run time: 56014 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174150 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f367e1c12510f72ed2667ae3b1b1a31229e90a24, data reload: false

query1	910	376	364	364
query2	6453	2551	2435	2435
query3	6639	207	215	207
query4	20055	17533	17235	17235
query5	3675	492	485	485
query6	270	177	170	170
query7	4606	304	301	301
query8	321	286	296	286
query9	8464	2376	2363	2363
query10	576	316	289	289
query11	10546	10034	10052	10034
query12	124	84	87	84
query13	1673	388	376	376
query14	10187	7245	7861	7245
query15	239	187	190	187
query16	7662	267	263	263
query17	1712	557	522	522
query18	1878	288	279	279
query19	208	160	157	157
query20	92	89	103	89
query21	210	122	127	122
query22	4247	4024	4004	4004
query23	33807	33660	33695	33660
query24	10944	2924	2896	2896
query25	642	413	380	380
query26	1160	164	163	163
query27	2259	326	340	326
query28	7141	2128	2110	2110
query29	897	630	636	630
query30	240	157	160	157
query31	971	791	745	745
query32	101	56	55	55
query33	766	300	289	289
query34	1025	492	505	492
query35	734	678	629	629
query36	1142	996	997	996
query37	151	81	82	81
query38	2983	2818	2844	2818
query39	878	868	832	832
query40	213	129	134	129
query41	54	65	50	50
query42	109	105	101	101
query43	611	552	553	552
query44	1263	736	754	736
query45	204	164	161	161
query46	1077	733	768	733
query47	1846	1736	1797	1736
query48	373	310	301	301
query49	843	407	406	406
query50	765	408	390	390
query51	6815	6805	6646	6646
query52	108	95	103	95
query53	377	299	298	298
query54	893	458	449	449
query55	78	77	74	74
query56	292	268	260	260
query57	1114	1034	1062	1034
query58	250	252	256	252
query59	3542	3292	3104	3104
query60	343	286	274	274
query61	94	89	89	89
query62	623	434	446	434
query63	325	305	290	290
query64	8813	2240	1761	1761
query65	3217	3095	3090	3090
query66	748	331	338	331
query67	15222	14937	14835	14835
query68	4847	559	549	549
query69	619	450	346	346
query70	1190	1146	1141	1141
query71	467	301	298	298
query72	7173	5532	5343	5343
query73	786	335	333	333
query74	5897	5499	5538	5499
query75	3700	2701	2686	2686
query76	3599	1018	907	907
query77	638	307	302	302
query78	10399	10092	9668	9668
query79	4571	507	518	507
query80	1169	482	480	480
query81	527	232	226	226
query82	722	108	110	108
query83	198	173	174	173
query84	274	95	88	88
query85	1374	281	274	274
query86	410	345	308	308
query87	3227	3197	3038	3038
query88	4501	2458	2512	2458
query89	491	386	378	378
query90	1871	195	192	192
query91	127	103	101	101
query92	62	53	48	48
query93	4179	517	513	513
query94	1095	189	192	189
query95	419	320	333	320
query96	611	266	277	266
query97	3205	2990	3008	2990
query98	227	205	206	205
query99	1165	849	856	849
Total cold run time: 276540 ms
Total hot run time: 174150 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f367e1c12510f72ed2667ae3b1b1a31229e90a24, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.22	0.04	0.05
query4	1.68	0.08	0.07
query5	0.48	0.47	0.50
query6	1.14	0.72	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.49	0.50
query10	0.54	0.55	0.55
query11	0.15	0.11	0.11
query12	0.15	0.12	0.13
query13	0.59	0.59	0.58
query14	0.77	0.82	0.78
query15	0.86	0.81	0.82
query16	0.36	0.36	0.37
query17	1.02	1.01	1.03
query18	0.21	0.26	0.22
query19	1.82	1.75	1.73
query20	0.01	0.01	0.01
query21	15.44	0.77	0.66
query22	4.39	7.38	1.78
query23	18.30	1.38	1.31
query24	2.13	0.22	0.21
query25	0.14	0.09	0.09
query26	0.27	0.18	0.17
query27	0.09	0.08	0.08
query28	13.33	1.01	0.99
query29	12.60	3.29	3.25
query30	0.24	0.06	0.05
query31	2.87	0.39	0.39
query32	3.26	0.49	0.46
query33	2.91	2.96	2.93
query34	16.95	4.42	4.36
query35	4.45	4.47	4.48
query36	0.65	0.47	0.47
query37	0.19	0.15	0.15
query38	0.16	0.15	0.15
query39	0.04	0.04	0.04
query40	0.18	0.15	0.15
query41	0.10	0.04	0.05
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.53 s
Total hot run time: 30.45 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jul 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jul 4, 2024

PR approved by anyone and no changes requested.

@morrySnow morrySnow merged commit 701c7db into apache:master Jul 5, 2024
26 of 29 checks passed
seawinde added a commit to seawinde/doris that referenced this pull request Jul 16, 2024
@seawinde
[improvement](mtmv) improve mv rewrite performance by reuse the shutt…
b2dad9a 
…led expression (apache#37197)

Optimizations:
1. Expression shuttle is expensive in materialized view rewritting, So
reuse the shuttled expression.
2. Generate shuttledExpressions by planOutput is also expensive, so
generate and store in struct info once and used later
@seawinde seawinde mentioned this pull request Jul 16, 2024
Merged
yiguolei pushed a commit that referenced this pull request Jul 16, 2024
@seawinde
[improvement](mtmv) improve mv rewrite performance by reuse the shutt…
dbd2b0a 
…led expression (#37197) (#37935)

## Proposed changes

chrry-pick 2.1
pr: #37197
commitId: 701c7db
dataroaring pushed a commit that referenced this pull request Jul 17, 2024
@seawinde @dataroaring
[improvement](mtmv) improve mv rewrite performance by reuse the shutt…
5b8e5ec 
…led expression (#37197)

Optimizations:
1. Expression shuttle is expensive in materialized view rewritting, So
reuse the shuttled expression.
2. Generate shuttledExpressions by planOutput is also expensive, so
generate and store in struct info once and used later
@gavinchou gavinchou mentioned this pull request Aug 19, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@morrySnow morrySnow morrySnow approved these changes

@keanji-x keanji-x keanji-x approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.5-merged dev/3.0.1-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@seawinde @doris-robot @keanji-x @morrySnow @yiguolei @dataroaring