Skip to content

[Improvement](materialized-view) forbidden mv rewriter when select stmt's from clause not have mv #27638

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
BiteTheDDDDt merged 3 commits into from
Nov 28, 2023
Merged

[Improvement](materialized-view) forbidden mv rewriter when select stmt's from clause not have mv #27638

BiteTheDDDDt merged 3 commits into from
Nov 28, 2023

Conversation

@BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented Nov 27, 2023

Proposed changes

forbidden mv rewriter when select stmt's from clause not have mv

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Nov 27, 2023

run buildall

morrySnow
morrySnow previously approved these changes Nov 27, 2023
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 27, 2023
@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2023

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 27, 2023

PR approved by anyone and no changes requested.

@wm1581066 wm1581066 added dev/2.0.3 usercase Important user case type label dev/2.0.4 and removed dev/2.0.3 labels Nov 27, 2023
@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Nov 27, 2023

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Nov 27, 2023
@doris-robot
Copy link

doris-robot commented Nov 27, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit 8d6a219e340e3227ff5b0c13345b445ebaf59058, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4954	4695	4689	4689
q2	378	175	158	158
q3	2163	1995	2031	1995
q4	1522	1386	1354	1354
q5	4073	4088	4057	4057
q6	278	135	133	133
q7	1442	889	898	889
q8	2965	2970	2968	2968
q9	9957	9946	9861	9861
q10	3476	3567	3595	3567
q11	390	252	255	252
q12	477	294	291	291
q13	4705	3841	3798	3798
q14	317	292	289	289
q15	585	532	533	532
q16	662	588	582	582
q17	1188	1037	981	981
q18	7889	7465	7424	7424
q19	1765	1765	1733	1733
q20	605	308	295	295
q21	4701	4254	4261	4254
q22	483	378	384	378
Total cold run time: 54975 ms
Total hot run time: 50480 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4578	4627	4608	4608
q2	366	235	234	234
q3	4079	4063	4060	4060
q4	2776	2761	2753	2753
q5	9705	9595	9681	9595
q6	283	126	121	121
q7	3073	2476	2476	2476
q8	4414	4386	4412	4386
q9	13272	13206	13100	13100
q10	4062	4167	4193	4167
q11	829	651	739	651
q12	990	823	798	798
q13	4470	3562	3572	3562
q14	399	352	350	350
q15	576	523	514	514
q16	743	673	683	673
q17	3827	3931	3818	3818
q18	9780	9272	9378	9272
q19	1904	1874	1885	1874
q20	2413	2093	2067	2067
q21	8985	8945	8889	8889
q22	956	809	794	794
Total cold run time: 82480 ms
Total hot run time: 78762 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2023

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G' 
Tpch sf100 test result on commit b1c966ca587e99d46e85b7f117765d80a9ea4c6c, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4919	4650	4653	4650
q2	365	149	158	149
q3	2020	1893	1840	1840
q4	1396	1296	1245	1245
q5	3960	3969	3987	3969
q6	258	138	132	132
q7	1414	889	885	885
q8	2783	2819	2777	2777
q9	9821	9710	9777	9710
q10	3469	3536	3519	3519
q11	389	240	236	236
q12	437	293	305	293
q13	4552	3797	3828	3797
q14	314	285	298	285
q15	574	543	519	519
q16	661	589	580	580
q17	1151	987	955	955
q18	7864	7530	7519	7519
q19	1703	1682	1678	1678
q20	546	298	309	298
q21	4456	3979	4037	3979
q22	477	377	382	377
Total cold run time: 53529 ms
Total hot run time: 49392 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4577	4553	4597	4553
q2	343	216	245	216
q3	4030	4023	4025	4023
q4	2726	2711	2717	2711
q5	9569	9588	9639	9588
q6	251	125	125	125
q7	3021	2495	2534	2495
q8	4448	4428	4422	4422
q9	13162	13075	13223	13075
q10	4070	4141	4158	4141
q11	757	659	644	644
q12	974	797	824	797
q13	4247	3548	3606	3548
q14	397	375	351	351
q15	575	518	521	518
q16	744	673	703	673
q17	3871	3863	3785	3785
q18	9582	9106	9181	9106
q19	1839	1782	1777	1777
q20	2427	2046	2050	2046
q21	8789	8420	8614	8420
q22	918	774	792	774
Total cold run time: 81317 ms
Total hot run time: 77788 ms

@doris-robot
Copy link

doris-robot commented Nov 27, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.31 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 28 seconds loaded 2358488459 Bytes, about 80 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.8 seconds inserted 10000000 Rows, about 347K ops/s
storage size: 17096971644 Bytes

@BiteTheDDDDt
Copy link
Contributor Author

BiteTheDDDDt commented Nov 28, 2023

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Nov 28, 2023

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Nov 28, 2023
@doris-robot
Copy link

doris-robot commented Nov 28, 2023

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 45.17 seconds
stream load tsv: 588 seconds loaded 74807831229 Bytes, about 121 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 34 seconds loaded 861443392 Bytes, about 24 MB/s
insert into select: 29.5 seconds inserted 10000000 Rows, about 338K ops/s
storage size: 17101354157 Bytes

@BiteTheDDDDt BiteTheDDDDt merged commit 31fe481 into apache:master Nov 28, 2023
XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
@BiteTheDDDDt
[Improvement](materialized-view) forbidden mv rewriter when select st…
e626df4 
…mt's from clause not have mv (apache#27638)

forbidden mv rewriter when select stmt's from clause not have mv
BiteTheDDDDt added a commit to BiteTheDDDDt/incubator-doris that referenced this pull request Dec 21, 2023
@BiteTheDDDDt
[Improvement](materialized-view) forbidden mv rewriter when select st…
312b9d5 
…mt's from clause not have mv (apache#27638)

forbidden mv rewriter when select stmt's from clause not have mv
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Dec 21, 2023
Merged
xiaokang pushed a commit that referenced this pull request Dec 22, 2023
@BiteTheDDDDt
[Improvement](materialized-view) forbidden mv rewriter when select st…
f75804e 
…mt's from clause not have mv (#27638) (#28798)
@BiteTheDDDDt BiteTheDDDDt deleted the dev_1127_2 branch January 20, 2025 06:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow left review comments

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/2.0.4-merged reviewed usercase Important user case type label

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@BiteTheDDDDt @doris-robot @Gabriel39 @starocean999 @morrySnow @xiaokang @wm1581066