Skip to content

[Opt](rf) Opt broadcast join remote runtime filter merge and wait#29439

Merged
yiguolei merged 1 commit intoapache:masterfrom
HappenLee:log
Jan 3, 2024
Merged

[Opt](rf) Opt broadcast join remote runtime filter merge and wait#29439
yiguolei merged 1 commit intoapache:masterfrom
HappenLee:log

Conversation

@HappenLee
Copy link
Contributor

Proposed changes

Before:

 RuntimeFilter:  (id  =  5,  type  =  bloomfilter):
                          -  Info:  [IsPushDown  =  true,  RuntimeFilterState  =  READY,  IsIgnored  =  false,  HasRemoteTarget  =  true,  HasLocalTarget  =  false]
                          -  EffectTime:  811  ms

After:

 RuntimeFilter:  (id  =  5,  type  =  bloomfilter):
                          -  Info:  [IsPushDown  =  true,  RuntimeFilterState  =  READY,  IsIgnored  =  false,  HasRemoteTarget  =  true,  HasLocalTarget  =  false]
                          -  EffectTime:  50  ms

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@HappenLee
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions


// publish runtime filter
Status publish() {
Status publish(bool publish_local = false) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: method 'publish' can be made static [readability-convert-member-functions-to-static]

Suggested change
Status publish(bool publish_local = false) {
static Status publish(bool publish_local = false) {

@doris-robot
Copy link

TPC-H test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G', run with scripts in https://github.com/apache/doris/tree/master/tools/tpch-tools

Tpch sf100 test result on commit 9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6, data reload: false

run tpch-sf100 query with default conf and session variables
q1	5486	5194	5169	5169
q2	400	195	158	158
q3	1476	1207	1309	1207
q4	1095	843	818	818
q5	3161	3127	3094	3094
q6	226	149	135	135
q7	994	575	534	534
q8	2134	2191	2238	2191
q9	6709	6699	6706	6699
q10	3200	3159	3109	3109
q11	367	221	220	220
q12	391	245	249	245
q13	4455	3677	3721	3677
q14	247	215	219	215
q15	605	552	534	534
q16	487	410	419	410
q17	1036	516	499	499
q18	7094	6783	6813	6783
q19	1655	1523	1468	1468
q20	565	352	353	352
q21	2933	2529	2465	2465
q22	408	311	326	311
Total cold run time: 45124 ms
Total hot run time: 40293 ms

run tpch-sf100 query with default conf and set session variable runtime_filter_mode=off
q1	5081	5054	5099	5054
q2	337	258	260	258
q3	3394	3332	3322	3322
q4	2160	2011	2036	2011
q5	6010	5981	5970	5970
q6	228	129	129	129
q7	2437	1978	1974	1974
q8	3575	3696	3700	3696
q9	9108	9045	9022	9022
q10	3926	3921	3927	3921
q11	621	500	478	478
q12	821	658	664	658
q13	3894	3240	3269	3240
q14	314	270	297	270
q15	613	553	541	541
q16	552	517	534	517
q17	2035	1848	1823	1823
q18	8764	8383	8779	8383
q19	1730	1674	1703	1674
q20	2336	2006	1982	1982
q21	5784	5435	5360	5360
q22	578	484	526	484
Total cold run time: 64298 ms
Total hot run time: 60767 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.61% (8613/23524)
Line Coverage: 28.67% (69990/244142)
Region Coverage: 27.65% (36227/131021)
Branch Coverage: 24.35% (18510/76014)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6_9fdcd3ee6f1470b40f83a0bc77b3ea2cae4342c6/report/index.html

@doris-robot
Copy link

(From new machine)TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 47.42 seconds
stream load tsv: 565 seconds loaded 74807831229 Bytes, about 126 MB/s
stream load json: 19 seconds loaded 2358488459 Bytes, about 118 MB/s
stream load orc: 66 seconds loaded 1101869774 Bytes, about 15 MB/s
stream load parquet: 32 seconds loaded 861443392 Bytes, about 25 MB/s
insert into select: 28.0 seconds inserted 10000000 Rows, about 357K ops/s
storage size: 17183460324 Bytes

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 3, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 3, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 14e7eb7 into apache:master Jan 3, 2024
HappenLee added a commit to HappenLee/incubator-doris that referenced this pull request Jan 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants