Skip to content

fix <=> runtime filter failed bug#32003

Merged
HappenLee merged 2 commits intoapache:masterfrom
HappenLee:guancheyun
Mar 11, 2024
Merged

fix <=> runtime filter failed bug#32003
HappenLee merged 2 commits intoapache:masterfrom
HappenLee:guancheyun

Conversation

@HappenLee
Copy link
Contributor

@HappenLee HappenLee commented Mar 8, 2024

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Mar 8, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@starocean999
Copy link
Contributor

starocean999 commented Mar 9, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 9, 2024

TeamCity be ut coverage result:
Function Coverage: 35.36% (8555/24194)
Line Coverage: 27.14% (69513/256097)
Region Coverage: 26.36% (36070/136847)
Branch Coverage: 23.20% (18435/79456)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d101e8c73852659a11d15d3f4e723b5635498939_d101e8c73852659a11d15d3f4e723b5635498939/report/index.html

@HappenLee
Copy link
Contributor Author

HappenLee commented Mar 10, 2024

run buildall

@HappenLee
Copy link
Contributor Author

HappenLee commented Mar 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TPC-H: Total hot run time: 37918 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713, data reload: false

------ Round 1 ----------------------------------
q1	17698	4083	4035	4035
q2	2022	150	141	141
q3	10561	925	925	925
q4	4662	940	940	940
q5	7609	2961	3001	2961
q6	181	123	126	123
q7	1306	838	822	822
q8	9564	2006	2068	2006
q9	7210	6504	6423	6423
q10	8239	2569	2564	2564
q11	417	217	219	217
q12	731	322	321	321
q13	17986	2932	2906	2906
q14	278	258	253	253
q15	493	455	456	455
q16	485	408	396	396
q17	956	882	818	818
q18	6815	5865	5785	5785
q19	1589	1531	1517	1517
q20	551	288	289	288
q21	7623	3735	3737	3735
q22	791	287	325	287
Total cold run time: 107767 ms
Total hot run time: 37918 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4117	3999	3988	3988
q2	321	224	230	224
q3	2955	2933	2921	2921
q4	1841	1805	1795	1795
q5	5230	5271	5262	5262
q6	202	115	117	115
q7	2283	1809	1828	1809
q8	3206	3274	3291	3274
q9	8511	8521	8535	8521
q10	6146	3733	3736	3733
q11	530	442	435	435
q12	689	576	562	562
q13	12497	2803	2759	2759
q14	279	250	272	250
q15	474	442	439	439
q16	449	397	406	397
q17	1707	1678	1669	1669
q18	7784	7521	7329	7329
q19	1801	1611	1616	1611
q20	1935	1684	1733	1684
q21	4910	4659	4779	4659
q22	528	416	407	407
Total cold run time: 68395 ms
Total hot run time: 53843 ms

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TPC-DS: Total hot run time: 187210 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713, data reload: false

query1	924	354	336	336
query2	7398	2144	2029	2029
query3	6714	213	213	213
query4	28254	21217	21131	21131
query5	4343	463	465	463
query6	277	186	171	171
query7	4633	331	301	301
query8	246	166	166	166
query9	8462	2287	2301	2287
query10	456	238	246	238
query11	15168	14589	14292	14292
query12	138	87	85	85
query13	1666	418	417	417
query14	10595	9932	10074	9932
query15	265	206	201	201
query16	7951	277	283	277
query17	1709	593	581	581
query18	2078	289	278	278
query19	203	161	180	161
query20	91	91	90	90
query21	203	131	130	130
query22	4682	4441	4449	4441
query23	31715	31082	31061	31061
query24	11150	3067	3089	3067
query25	671	395	390	390
query26	1009	172	169	169
query27	2769	362	365	362
query28	6290	1868	1871	1868
query29	1047	610	608	608
query30	310	151	152	151
query31	958	753	757	753
query32	106	68	65	65
query33	760	264	262	262
query34	1018	476	476	476
query35	843	646	642	642
query36	962	814	826	814
query37	118	84	80	80
query38	3316	3139	3098	3098
query39	1402	1362	1360	1360
query40	217	117	115	115
query41	62	55	54	54
query42	109	102	105	102
query43	442	410	394	394
query44	1094	728	709	709
query45	297	261	265	261
query46	1059	785	794	785
query47	1614	1560	1563	1560
query48	445	354	351	351
query49	1142	343	335	335
query50	779	381	389	381
query51	6681	6576	6597	6576
query52	115	109	86	86
query53	346	290	283	283
query54	285	243	243	243
query55	88	86	90	86
query56	228	209	217	209
query57	1069	991	988	988
query58	231	233	217	217
query59	2549	2462	2296	2296
query60	255	284	272	272
query61	95	92	90	90
query62	668	422	414	414
query63	323	282	278	278
query64	4890	3102	3112	3102
query65	3035	2997	2986	2986
query66	828	340	331	331
query67	14964	14352	14591	14352
query68	11190	585	584	584
query69	731	385	394	385
query70	1361	1127	1144	1127
query71	579	285	270	270
query72	10077	2559	2483	2483
query73	1765	339	343	339
query74	7243	6760	6771	6760
query75	11031	8047	8043	8043
query76	7321	1118	1189	1118
query77	705	266	262	262
query78	10154	9702	9481	9481
query79	9117	531	526	526
query80	2287	418	442	418
query81	499	215	208	208
query82	377	200	201	200
query83	252	155	154	154
query84	286	81	80	80
query85	1302	395	369	369
query86	374	304	281	281
query87	3349	3237	3137	3137
query88	3239	2385	2370	2370
query89	520	362	404	362
query90	2440	184	181	181
query91	165	129	129	129
query92	60	51	55	51
query93	4797	538	538	538
query94	1702	193	190	190
query95	475	361	355	355
query96	598	281	270	270
query97	4477	4263	4299	4263
query98	246	219	221	219
query99	1121	755	757	755
Total cold run time: 314650 ms
Total hot run time: 187210 ms

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713, data reload: false

query1	0.05	0.04	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.10
query5	0.52	0.51	0.48
query6	1.26	0.65	0.67
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.51
query10	0.58	0.55	0.57
query11	0.14	0.10	0.11
query12	0.12	0.11	0.10
query13	0.58	0.57	0.56
query14	0.74	0.75	0.76
query15	0.83	0.80	0.80
query16	0.37	0.37	0.37
query17	1.00	0.95	0.96
query18	0.26	0.25	0.24
query19	1.79	1.73	1.75
query20	0.01	0.02	0.01
query21	15.41	0.66	0.59
query22	3.25	4.95	2.75
query23	17.57	1.07	0.91
query24	2.37	0.26	0.26
query25	0.21	0.10	0.05
query26	0.18	0.14	0.14
query27	0.04	0.03	0.03
query28	12.31	0.84	0.82
query29	12.53	3.24	3.19
query30	0.63	0.55	0.58
query31	2.80	0.34	0.36
query32	3.34	0.44	0.43
query33	2.97	2.97	2.92
query34	15.50	4.33	4.26
query35	4.29	4.33	4.30
query36	1.09	1.01	1.02
query37	0.07	0.06	0.06
query38	0.05	0.03	0.03
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.08	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.76 s
Total hot run time: 30.9 s

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       16.1 seconds inserted 10000000 Rows, about 621K ops/s

@doris-robot
Copy link

doris-robot commented Mar 10, 2024

TeamCity be ut coverage result:
Function Coverage: 35.35% (8555/24199)
Line Coverage: 27.14% (69515/256144)
Region Coverage: 26.36% (36064/136839)
Branch Coverage: 23.21% (18437/79446)
Coverage Report: http://coverage.selectdb-in.cc/coverage/9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713_9fdc9bcb11c7f5eab6b5e11b7859c4e8280a4713/report/index.html

@HappenLee HappenLee marked this pull request as ready for review March 10, 2024 09:51
wm1581066
wm1581066 approved these changes Mar 10, 2024
View reviewed changes
Copy link

@wm1581066 wm1581066 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Mar 10, 2024

PR approved by anyone and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Mar 11, 2024

PR approved by at least one committer and no changes requested.

@HappenLee HappenLee merged commit 784e570 into apache:master Mar 11, 2024
yiguolei pushed a commit that referenced this pull request Mar 12, 2024
@HappenLee @Doris-Extras
[RuntimeFilter] fix <=> runtime filter failed bug (#32003)
2470634
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Gabriel39 Gabriel39 Gabriel39 approved these changes

+1 more reviewer

@wm1581066 wm1581066 wm1581066 approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. meta-change reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@HappenLee @doris-robot @starocean999 @Gabriel39 @wm1581066