Skip to content

Conversation

@Gabriel39
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@Gabriel39 Gabriel39 changed the title [pipelineX](local shuffle) Use local shuffle to optimize BHJ [WIP][pipelineX](local shuffle) Use local shuffle to optimize BHJ Nov 30, 2023
@Gabriel39
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

return _sub_plan_query_statistics_recvr;
}

bool need_to_local_shuffle() const override {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'need_to_local_shuffle' can be made static [readability-convert-member-functions-to-static]

Suggested change
bool need_to_local_shuffle() const override {
static bool need_to_local_shuffle() override {

return Status::OK();
}

Status PipelineXFragmentContext::_plan_local_shuffle() {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method '_plan_local_shuffle' can be made static [readability-convert-member-functions-to-static]

be/src/pipeline/pipeline_x/pipeline_x_fragment_context.h:155:

-     Status _plan_local_shuffle();
+     static Status _plan_local_shuffle();

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 44.27 seconds
stream load tsv: 559 seconds loaded 74807831229 Bytes, about 127 MB/s
stream load json: 18 seconds loaded 2358488459 Bytes, about 124 MB/s
stream load orc: 65 seconds loaded 1101869774 Bytes, about 16 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.2 seconds inserted 10000000 Rows, about 354K ops/s
storage size: 17164300496 Bytes

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Nov 30, 2023
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@yiguolei yiguolei left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@yiguolei yiguolei merged commit c0aac04 into apache:master Nov 30, 2023
@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
Tpch sf100 test result on commit 71e01f3a0d541121e1079ed90b0f256ead67dfd2, data reload: false

run tpch-sf100 query with default conf and session variables
q1	4884	4616	4670	4616
q2	353	161	158	158
q3	1515	1328	1283	1283
q4	1141	960	962	960
q5	3247	3213	3242	3213
q6	252	135	136	135
q7	986	518	552	518
q8	2295	2249	2229	2229
q9	6971	6968	6918	6918
q10	3305	3352	3365	3352
q11	341	205	199	199
q12	347	206	218	206
q13	4764	3902	3882	3882
q14	244	212	214	212
q15	579	540	519	519
q16	443	402	395	395
q17	1045	679	646	646
q18	7994	7415	7825	7415
q19	1579	1545	1505	1505
q20	596	336	314	314
q21	3425	2978	3011	2978
q22	364	298	305	298
Total cold run time: 46670 ms
Total hot run time: 41951 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	4605	4589	4578	4578
q2	314	182	224	182
q3	3743	3743	3750	3743
q4	2538	2502	2500	2500
q5	6184	6179	6179	6179
q6	243	119	120	119
q7	2586	1976	1951	1951
q8	3751	3681	3645	3645
q9	9438	9397	9413	9397
q10	4071	4134	4156	4134
q11	635	506	523	506
q12	798	620	632	620
q13	4383	3656	3599	3599
q14	270	258	245	245
q15	584	528	517	517
q16	510	503	489	489
q17	2061	2074	2048	2048
q18	9536	8786	9737	8786
q19	1784	1790	1791	1790
q20	2319	2003	1974	1974
q21	7311	6980	6913	6913
q22	676	549	552	549
Total cold run time: 68340 ms
Total hot run time: 64464 ms

XuJianxu pushed a commit to XuJianxu/doris that referenced this pull request Dec 14, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants