Skip to content

[opt](Nereids) derive physical properties of Project and Filter#29171

Merged
englefly merged 1 commit intoapache:masterfrom
morrySnow:add_filter_project_diriver
Dec 28, 2023
Merged

[opt](Nereids) derive physical properties of Project and Filter#29171
englefly merged 1 commit intoapache:masterfrom
morrySnow:add_filter_project_diriver

Conversation

@morrySnow
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@morrySnow
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.79 seconds
stream load tsv: 567 seconds loaded 74807831229 Bytes, about 125 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.1 seconds inserted 10000000 Rows, about 355K ops/s
storage size: 17184206126 Bytes

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Tpch sf100 test result on commit 9c78f192fc9dfb278b14f0491cb96cc0763dd24b, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4986	4681	4659	4659
q2	370	144	152	144
q3	1478	1314	1293	1293
q4	1143	936	885	885
q5	3193	3180	3163	3163
q6	249	130	129	129
q7	1021	505	509	505
q8	2284	2279	2273	2273
q9	6617	6596	6574	6574
q10	3202	3265	3272	3265
q11	336	197	210	197
q12	350	202	212	202
q13	4144	3431	3407	3407
q14	241	213	218	213
q15	574	526	520	520
q16	442	384	389	384
q17	1043	805	640	640
q18	7088	6806	6797	6797
q19	1642	1644	1636	1636
q20	533	328	300	300
q21	3207	2742	2709	2709
q22	374	311	309	309
Total cold run time: 44517 ms
Total hot run time: 40204 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4603	4611	4620	4611
q2	273	167	164	164
q3	3378	3366	3372	3366
q4	2236	2224	2219	2219
q5	5734	5731	5716	5716
q6	240	119	117	117
q7	2367	1870	1839	1839
q8	3648	3629	3656	3629
q9	9059	9011	8985	8985
q10	3794	3877	3899	3877
q11	480	383	379	379
q12	767	598	586	586
q13	3935	3231	3173	3173
q14	288	243	260	243
q15	566	520	526	520
q16	502	447	483	447
q17	1990	1948	1959	1948
q18	8627	8087	8305	8087
q19	1764	1774	1774	1774
q20	2242	1951	1908	1908
q21	6176	5779	5810	5779
q22	554	457	454	454
Total cold run time: 63223 ms
Total hot run time: 59821 ms

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 27, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morrySnow morrySnow marked this pull request as ready for review December 28, 2023 12:04
--------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF4 i_item_sk->[ss_item_sk]
----------------PhysicalProject
------------------hashJoin[INNER_JOIN] hashCondition=((store_sales.ss_item_sk = item.i_item_sk)) otherCondition=() build RFs:RF4 i_item_sk->[ss_item_sk]
--------------------PhysicalDistribute
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ds1t will have 20% perf drop for this bc->shuffle, mark�

@englefly englefly merged commit c3679a2 into apache:master Dec 28, 2023
HappenLee pushed a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
@morrySnow morrySnow deleted the add_filter_project_diriver branch February 27, 2024 10:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants