Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug](runtime-filter) make need_local_merge unrelated with broadcast and support merge on bitmap_filter #33636

Merged
merged 4 commits into from
Apr 16, 2024
Merged

[Bug](runtime-filter) make need_local_merge unrelated with broadcast and support merge on bitmap_filter #33636

merged 4 commits into from
Apr 16, 2024

Conversation

BiteTheDDDDt
Copy link
Contributor

Proposed changes

registe consumer rf in local when rf is broadcast

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.59% (8905/25020)
Line Coverage: 27.32% (73148/267748)
Region Coverage: 26.45% (37833/143029)
Branch Coverage: 23.21% (19277/83072)
Coverage Report: http://coverage.selectdb-in.cc/coverage/4bf5e14299741f42ca4e9a77685a2a0c01e99af0_4bf5e14299741f42ca4e9a77685a2a0c01e99af0/report/index.html

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.59% (8905/25020)
Line Coverage: 27.31% (73130/267754)
Region Coverage: 26.44% (37823/143027)
Branch Coverage: 23.19% (19266/83070)
Coverage Report: http://coverage.selectdb-in.cc/coverage/0e93f9889ce84fc4c213a686a5565297d05f8d32_0e93f9889ce84fc4c213a686a5565297d05f8d32/report/index.html

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@BiteTheDDDDt BiteTheDDDDt changed the title [Bug](runtime-filter) registe consumer rf in local when rf is broadcast [Bug](runtime-filter) make need_local_merge unrelated with broadcast and support merge on bitmap_filter Apr 15, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.56% (8905/25044)
Line Coverage: 27.29% (73134/267959)
Region Coverage: 26.43% (37827/143130)
Branch Coverage: 23.18% (19271/83120)
Coverage Report: http://coverage.selectdb-in.cc/coverage/083166bb1d4de65504f54b67dbf01e90d4508f9c_083166bb1d4de65504f54b67dbf01e90d4508f9c/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 38391 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 083166bb1d4de65504f54b67dbf01e90d4508f9c, data reload: false

------ Round 1 ----------------------------------
q1	17610	4376	4233	4233
q2	2016	194	195	194
q3	10421	1225	1224	1224
q4	10195	711	817	711
q5	7520	2680	2655	2655
q6	216	130	130	130
q7	1020	594	570	570
q8	9218	2077	2038	2038
q9	7905	6579	6548	6548
q10	8587	3531	3509	3509
q11	458	250	225	225
q12	453	221	214	214
q13	18847	2923	2905	2905
q14	269	231	229	229
q15	508	484	476	476
q16	537	377	373	373
q17	973	692	728	692
q18	7402	6744	6598	6598
q19	6942	1506	1480	1480
q20	674	303	297	297
q21	3487	2792	2791	2791
q22	360	299	307	299
Total cold run time: 115618 ms
Total hot run time: 38391 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4355	4204	4232	4204
q2	377	264	264	264
q3	2989	2743	2747	2743
q4	1837	1606	1582	1582
q5	5307	5296	5341	5296
q6	209	123	123	123
q7	2246	1878	1881	1878
q8	3233	3381	3375	3375
q9	8580	8561	8775	8561
q10	4080	3883	4029	3883
q11	606	484	469	469
q12	822	631	655	631
q13	16000	3209	3203	3203
q14	325	281	286	281
q15	522	477	507	477
q16	505	428	442	428
q17	1837	1605	1534	1534
q18	8083	7910	7736	7736
q19	1648	1552	1632	1552
q20	2039	1899	1803	1803
q21	5037	4950	4931	4931
q22	542	488	453	453
Total cold run time: 71179 ms
Total hot run time: 55407 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185667 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 083166bb1d4de65504f54b67dbf01e90d4508f9c, data reload: false

query1	888	1121	1128	1121
query2	6180	2590	2401	2401
query3	6646	204	199	199
query4	32670	21409	21403	21403
query5	4124	408	412	408
query6	248	188	174	174
query7	4024	287	286	286
query8	222	173	174	173
query9	5625	2300	2279	2279
query10	366	233	238	233
query11	14757	14215	14289	14215
query12	139	89	85	85
query13	996	359	397	359
query14	9327	7127	6847	6847
query15	222	176	190	176
query16	7227	251	258	251
query17	1806	592	552	552
query18	1786	281	265	265
query19	192	153	149	149
query20	96	87	85	85
query21	202	125	124	124
query22	4960	4849	4831	4831
query23	33790	33049	33222	33049
query24	11370	3037	3008	3008
query25	584	428	381	381
query26	733	162	158	158
query27	3015	359	380	359
query28	6733	2100	2106	2100
query29	877	650	619	619
query30	285	186	173	173
query31	945	741	779	741
query32	71	51	53	51
query33	546	252	264	252
query34	946	512	511	511
query35	865	736	697	697
query36	1027	940	913	913
query37	115	69	68	68
query38	3427	3292	3310	3292
query39	1622	1585	1582	1582
query40	175	129	134	129
query41	46	42	45	42
query42	100	96	94	94
query43	573	538	553	538
query44	1282	729	734	729
query45	266	272	280	272
query46	1087	767	719	719
query47	2021	1953	1966	1953
query48	382	296	307	296
query49	840	397	384	384
query50	788	398	392	392
query51	6867	6778	6760	6760
query52	96	88	88	88
query53	348	274	282	274
query54	249	225	233	225
query55	72	72	69	69
query56	241	221	229	221
query57	1171	1126	1120	1120
query58	216	200	199	199
query59	3256	3349	3415	3349
query60	247	230	239	230
query61	94	90	88	88
query62	605	442	437	437
query63	301	280	283	280
query64	4296	4055	3930	3930
query65	3069	3033	3004	3004
query66	738	326	333	326
query67	15373	15014	14910	14910
query68	6963	548	564	548
query69	556	326	308	308
query70	1254	1182	1107	1107
query71	449	281	269	269
query72	6600	2759	2561	2561
query73	832	330	324	324
query74	6910	6385	6374	6374
query75	3424	2662	2664	2662
query76	4142	1108	1121	1108
query77	634	249	262	249
query78	10913	10226	10154	10154
query79	9784	512	520	512
query80	2211	456	466	456
query81	506	232	229	229
query82	1117	103	101	101
query83	209	173	169	169
query84	268	85	88	85
query85	1067	317	319	317
query86	452	311	325	311
query87	3423	3207	3227	3207
query88	6217	2279	2298	2279
query89	537	383	378	378
query90	2119	177	181	177
query91	123	97	95	95
query92	64	46	46	46
query93	6901	508	499	499
query94	1145	175	178	175
query95	1090	1107	1100	1100
query96	616	258	258	258
query97	3151	2908	2925	2908
query98	243	215	217	215
query99	1237	883	858	858
Total cold run time: 299610 ms
Total hot run time: 185667 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.11 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 083166bb1d4de65504f54b67dbf01e90d4508f9c, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.22	0.05	0.06
query4	1.68	0.06	0.09
query5	0.48	0.48	0.48
query6	1.48	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.05
query9	0.56	0.49	0.49
query10	0.53	0.56	0.55
query11	0.16	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.59	0.58
query14	0.76	0.77	0.76
query15	0.83	0.81	0.80
query16	0.37	0.37	0.40
query17	1.01	0.99	0.98
query18	0.22	0.24	0.22
query19	1.86	1.79	1.74
query20	0.02	0.01	0.01
query21	15.39	0.65	0.65
query22	4.09	7.12	1.74
query23	18.31	1.31	1.29
query24	1.98	0.21	0.22
query25	0.14	0.07	0.07
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.37	1.00	0.98
query29	12.62	3.21	3.22
query30	0.26	0.06	0.06
query31	2.86	0.38	0.37
query32	3.30	0.46	0.46
query33	2.81	2.82	2.85
query34	17.15	4.37	4.42
query35	4.48	4.45	4.46
query36	0.64	0.44	0.46
query37	0.18	0.15	0.15
query38	0.15	0.14	0.15
query39	0.04	0.04	0.04
query40	0.17	0.13	0.14
query41	0.09	0.06	0.06
query42	0.05	0.06	0.04
query43	0.05	0.03	0.03
Total cold run time: 109.57 s
Total hot run time: 30.11 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 083166bb1d4de65504f54b67dbf01e90d4508f9c with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.2 seconds inserted 10000000 Rows, about 757K ops/s

luozenglin
luozenglin approved these changes Apr 16, 2024
View reviewed changes
Copy link
Contributor

@luozenglin luozenglin left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 9e091cf into apache:master Apr 16, 2024
26 of 30 checks passed
@BiteTheDDDDt BiteTheDDDDt mentioned this pull request Apr 16, 2024
Merged
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@BiteTheDDDDt @dataroaring
[Bug](runtime-filter) make need_local_merge unrelated with broadcast …
ed39f68 
…and support merge on bitmap_filter (#33636)

make need_local_merge unrelated with broadcast and support merge on bitmap_filter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@luozenglin luozenglin luozenglin approved these changes

@zclllyybb zclllyybb zclllyybb approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.1.3-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@BiteTheDDDDt @doris-robot @luozenglin @zclllyybb @yiguolei