Skip to content

Conversation

@BiteTheDDDDt
Copy link
Contributor

@BiteTheDDDDt BiteTheDDDDt commented May 26, 2024

Proposed changes

merge from #35064 and #34914

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

return cast_to_string<type, ValueType>(v.get_value(), v.get_scale());
}

bool RuntimePredicate::_init(PrimitiveType type) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_init' exceeds recommended size/complexity thresholds [readability-function-size]

bool RuntimePredicate::_init(PrimitiveType type) {
                       ^
Additional context

be/src/runtime/runtime_predicate.cpp:111: 82 lines including whitespace and comments (threshold 80)

bool RuntimePredicate::_init(PrimitiveType type) {
                       ^


#pragma once

#include <gen_cpp/types.pb.h>
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: 'gen_cpp/types.pb.h' file not found [clang-diagnostic-error]

#include <gen_cpp/types.pb.h>
         ^

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.63% (9014/25299)
Line Coverage: 27.31% (74546/272962)
Region Coverage: 26.52% (38554/145375)
Branch Coverage: 23.39% (19659/84054)
Coverage Report: http://coverage.selectdb-in.cc/coverage/facb1942ef8d3e5aa89fd15523b80da80fa8afd3_facb1942ef8d3e5aa89fd15523b80da80fa8afd3/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41148 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit facb1942ef8d3e5aa89fd15523b80da80fa8afd3, data reload: false

------ Round 1 ----------------------------------
q1	17609	4285	4296	4285
q2	2018	187	199	187
q3	10467	1230	1213	1213
q4	10198	742	892	742
q5	7464	2755	2739	2739
q6	230	135	140	135
q7	970	617	606	606
q8	9231	2140	2087	2087
q9	10037	6643	6684	6643
q10	9325	4006	3933	3933
q11	443	244	229	229
q12	461	223	221	221
q13	18415	3277	3119	3119
q14	261	227	214	214
q15	522	466	500	466
q16	521	397	401	397
q17	970	735	771	735
q18	8436	7813	7962	7813
q19	7390	1579	1530	1530
q20	648	325	322	322
q21	5128	3263	3921	3263
q22	348	269	285	269
Total cold run time: 121092 ms
Total hot run time: 41148 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4719	4423	4386	4386
q2	364	275	274	274
q3	3214	2913	2985	2913
q4	1859	1630	1636	1630
q5	5546	5510	5524	5510
q6	212	122	125	122
q7	2170	1840	1799	1799
q8	3252	3415	3382	3382
q9	8642	8632	8658	8632
q10	3995	3703	3817	3703
q11	588	502	501	501
q12	784	628	646	628
q13	17051	3163	3201	3163
q14	298	270	280	270
q15	528	481	480	480
q16	499	436	458	436
q17	1806	1492	1467	1467
q18	7594	7582	7401	7401
q19	1674	1553	1524	1524
q20	1980	1791	1800	1791
q21	8316	4706	4669	4669
q22	554	480	489	480
Total cold run time: 75645 ms
Total hot run time: 55161 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172205 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit facb1942ef8d3e5aa89fd15523b80da80fa8afd3, data reload: false

query1	915	382	369	369
query2	6434	2403	2277	2277
query3	6661	209	212	209
query4	21510	17348	17323	17323
query5	4107	419	408	408
query6	245	155	165	155
query7	4599	300	301	300
query8	238	176	182	176
query9	8532	2416	2368	2368
query10	449	273	261	261
query11	10630	10134	10121	10121
query12	144	89	90	89
query13	1656	372	370	370
query14	10215	7033	7713	7033
query15	247	166	174	166
query16	8006	277	272	272
query17	1685	545	544	544
query18	2068	277	288	277
query19	203	160	206	160
query20	90	85	83	83
query21	201	137	130	130
query22	4154	3919	3840	3840
query23	33612	32971	33199	32971
query24	10932	2882	2863	2863
query25	559	355	372	355
query26	703	157	156	156
query27	2209	324	329	324
query28	6029	2071	2065	2065
query29	861	613	592	592
query30	216	152	153	152
query31	948	783	753	753
query32	89	53	56	53
query33	739	275	265	265
query34	917	488	485	485
query35	757	611	609	609
query36	1082	916	906	906
query37	106	67	66	66
query38	2875	2789	2782	2782
query39	852	799	831	799
query40	205	121	124	121
query41	46	43	45	43
query42	102	98	98	98
query43	601	546	548	546
query44	1222	737	742	737
query45	176	162	181	162
query46	1068	742	733	733
query47	1846	1761	1775	1761
query48	373	302	305	302
query49	846	388	381	381
query50	761	402	391	391
query51	6934	6814	6711	6711
query52	108	90	93	90
query53	358	298	302	298
query54	860	428	431	428
query55	73	78	73	73
query56	269	243	242	242
query57	1122	1047	1020	1020
query58	232	212	209	209
query59	3285	3218	3244	3218
query60	282	289	256	256
query61	102	93	88	88
query62	622	458	446	446
query63	315	294	293	293
query64	8521	2188	1696	1696
query65	3171	3080	3132	3080
query66	775	332	370	332
query67	15323	14922	14805	14805
query68	6444	562	556	556
query69	519	275	278	275
query70	1186	1193	1143	1143
query71	477	277	269	269
query72	8261	5360	5636	5360
query73	767	333	326	326
query74	6066	5563	5620	5563
query75	4424	2633	2655	2633
query76	3967	928	1059	928
query77	738	272	265	265
query78	10228	9790	9648	9648
query79	3245	525	510	510
query80	1077	444	441	441
query81	485	219	231	219
query82	759	95	93	93
query83	196	174	171	171
query84	268	92	85	85
query85	1267	304	258	258
query86	391	308	298	298
query87	3310	3100	3122	3100
query88	4185	2353	2350	2350
query89	486	384	403	384
query90	2045	196	185	185
query91	120	96	97	96
query92	60	50	47	47
query93	4898	536	515	515
query94	1052	186	182	182
query95	414	308	310	308
query96	599	265	262	262
query97	3212	3014	2998	2998
query98	234	216	211	211
query99	1222	866	839	839
Total cold run time: 278675 ms
Total hot run time: 172205 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.98 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit facb1942ef8d3e5aa89fd15523b80da80fa8afd3, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.06	0.05
query4	1.66	0.08	0.09
query5	0.48	0.51	0.52
query6	1.12	0.73	0.74
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.48	0.50
query10	0.55	0.56	0.53
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.58	0.59	0.60
query14	0.76	0.77	0.79
query15	0.83	0.82	0.81
query16	0.36	0.37	0.35
query17	0.97	1.03	1.06
query18	0.21	0.24	0.26
query19	1.82	1.84	1.76
query20	0.02	0.01	0.01
query21	15.74	0.65	0.64
query22	4.62	6.96	2.28
query23	18.30	1.46	1.31
query24	1.67	0.30	0.22
query25	0.15	0.09	0.09
query26	0.26	0.16	0.16
query27	0.08	0.08	0.08
query28	13.36	1.03	0.99
query29	13.00	3.29	3.23
query30	0.24	0.06	0.07
query31	2.86	0.37	0.39
query32	3.28	0.46	0.49
query33	2.91	2.85	2.90
query34	17.19	4.46	4.43
query35	4.53	4.48	4.47
query36	0.67	0.53	0.49
query37	0.19	0.17	0.15
query38	0.14	0.14	0.15
query39	0.05	0.03	0.03
query40	0.16	0.14	0.13
query41	0.09	0.04	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 110.23 s
Total hot run time: 30.98 s

@BiteTheDDDDt
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.64% (9016/25299)
Line Coverage: 27.32% (74563/272965)
Region Coverage: 26.53% (38565/145378)
Branch Coverage: 23.39% (19662/84056)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a890723156406c5e9b1a7a3a03321c5b2534153f_a890723156406c5e9b1a7a3a03321c5b2534153f/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 41476 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a890723156406c5e9b1a7a3a03321c5b2534153f, data reload: false

------ Round 1 ----------------------------------
q1	17625	4349	4207	4207
q2	2023	184	190	184
q3	10454	1258	1261	1258
q4	10187	786	774	774
q5	7469	2699	2663	2663
q6	227	136	127	127
q7	981	612	600	600
q8	9228	2076	2081	2076
q9	9706	7126	7113	7113
q10	9282	3857	3925	3857
q11	455	243	231	231
q12	488	246	226	226
q13	17375	3154	3341	3154
q14	273	214	223	214
q15	504	469	465	465
q16	491	387	389	387
q17	967	742	758	742
q18	8356	7849	7888	7849
q19	4447	1542	1558	1542
q20	639	327	311	311
q21	5199	3232	3956	3232
q22	349	264	280	264
Total cold run time: 116725 ms
Total hot run time: 41476 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4575	4443	4414	4414
q2	373	260	257	257
q3	3146	2972	2979	2972
q4	1845	1558	1605	1558
q5	5461	5524	5485	5485
q6	220	124	130	124
q7	2197	1805	1823	1805
q8	3284	3381	3385	3381
q9	9037	9107	9113	9107
q10	3937	3823	3839	3823
q11	612	507	507	507
q12	809	633	641	633
q13	16905	3135	3229	3135
q14	310	268	278	268
q15	534	494	477	477
q16	475	430	429	429
q17	1766	1486	1491	1486
q18	7706	7624	7320	7320
q19	1654	1548	1537	1537
q20	1998	1755	1788	1755
q21	13073	4707	4750	4707
q22	559	484	489	484
Total cold run time: 80476 ms
Total hot run time: 55664 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 172201 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a890723156406c5e9b1a7a3a03321c5b2534153f, data reload: false

query1	933	382	390	382
query2	6716	2418	2277	2277
query3	6645	215	213	213
query4	19608	17411	17162	17162
query5	4142	426	434	426
query6	244	175	154	154
query7	4572	303	294	294
query8	242	190	182	182
query9	8371	2406	2378	2378
query10	449	282	271	271
query11	10818	10063	10102	10063
query12	139	91	89	89
query13	1654	365	378	365
query14	9295	6888	7591	6888
query15	212	172	172	172
query16	7726	282	272	272
query17	1288	536	541	536
query18	1865	282	281	281
query19	212	163	161	161
query20	95	87	86	86
query21	198	135	132	132
query22	4173	3990	3926	3926
query23	33640	33176	33216	33176
query24	6998	2873	2941	2873
query25	549	351	364	351
query26	713	154	158	154
query27	2044	325	317	317
query28	3941	2099	2083	2083
query29	904	601	601	601
query30	230	150	157	150
query31	982	766	753	753
query32	83	50	57	50
query33	495	271	267	267
query34	873	475	475	475
query35	714	596	597	596
query36	1070	923	922	922
query37	100	67	82	67
query38	2894	2776	2783	2776
query39	844	786	808	786
query40	196	127	126	126
query41	45	43	44	43
query42	101	94	96	94
query43	579	537	551	537
query44	1068	727	746	727
query45	177	162	165	162
query46	1072	740	746	740
query47	1863	1767	1793	1767
query48	376	286	299	286
query49	767	376	386	376
query50	775	396	374	374
query51	6878	6642	6693	6642
query52	106	87	92	87
query53	351	281	288	281
query54	549	426	421	421
query55	77	73	73	73
query56	273	238	246	238
query57	1090	1034	1052	1034
query58	228	205	209	205
query59	3437	3164	3317	3164
query60	273	258	274	258
query61	92	89	85	85
query62	561	465	454	454
query63	314	280	283	280
query64	8444	2254	1762	1762
query65	3194	3089	3158	3089
query66	794	327	320	320
query67	15214	15115	14766	14766
query68	5396	539	559	539
query69	490	265	269	265
query70	1195	1153	1076	1076
query71	464	265	273	265
query72	8166	5311	5390	5311
query73	767	327	322	322
query74	5993	5651	5665	5651
query75	3633	2672	2625	2625
query76	3258	960	1011	960
query77	578	279	268	268
query78	10271	9829	9898	9829
query79	2050	533	518	518
query80	1460	443	432	432
query81	506	241	222	222
query82	1290	92	89	89
query83	255	177	180	177
query84	254	86	82	82
query85	1123	263	268	263
query86	459	302	328	302
query87	3299	3160	3144	3144
query88	4029	2369	2331	2331
query89	486	381	373	373
query90	2145	185	183	183
query91	125	95	96	95
query92	62	50	49	49
query93	2599	516	508	508
query94	1182	187	184	184
query95	399	319	298	298
query96	605	265	263	263
query97	3228	3018	3027	3018
query98	265	221	213	213
query99	1180	872	848	848
Total cold run time: 262777 ms
Total hot run time: 172201 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.56 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a890723156406c5e9b1a7a3a03321c5b2534153f, data reload: false

query1	0.04	0.03	0.03
query2	0.07	0.04	0.04
query3	0.23	0.05	0.05
query4	1.68	0.07	0.08
query5	0.48	0.50	0.50
query6	1.12	0.72	0.72
query7	0.02	0.02	0.02
query8	0.05	0.04	0.05
query9	0.54	0.48	0.49
query10	0.54	0.56	0.54
query11	0.16	0.12	0.12
query12	0.15	0.12	0.12
query13	0.59	0.59	0.60
query14	0.78	0.80	0.77
query15	0.84	0.81	0.81
query16	0.36	0.37	0.36
query17	0.94	1.00	0.96
query18	0.20	0.22	0.28
query19	1.81	1.69	1.68
query20	0.02	0.01	0.01
query21	15.74	0.65	0.64
query22	4.41	7.55	1.79
query23	18.29	1.33	1.26
query24	1.31	0.32	0.30
query25	0.14	0.08	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.08
query28	13.40	1.09	1.09
query29	13.03	3.30	3.27
query30	0.24	0.05	0.05
query31	2.88	0.38	0.39
query32	3.29	0.46	0.47
query33	2.86	2.88	2.90
query34	17.18	4.45	4.48
query35	4.50	4.67	4.50
query36	0.65	0.47	0.45
query37	0.17	0.16	0.16
query38	0.14	0.14	0.14
query39	0.04	0.04	0.03
query40	0.16	0.14	0.15
query41	0.10	0.05	0.05
query42	0.05	0.04	0.04
query43	0.04	0.04	0.03
Total cold run time: 109.58 s
Total hot run time: 30.56 s

@BiteTheDDDDt BiteTheDDDDt changed the title Test 0526 [Feature](topn) support general expr with topn filter and some refactor May 27, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 27, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@BiteTheDDDDt BiteTheDDDDt merged commit 99c5a80 into apache:master May 27, 2024
dataroaring pushed a commit that referenced this pull request May 27, 2024
…or (#35405)

support general expr with topn filter and some refactor
Co-authored-by: minghong <englefly@gmail.com>
seawinde pushed a commit to seawinde/doris that referenced this pull request May 27, 2024
…or (apache#35405)

support general expr with topn filter and some refactor
Co-authored-by: minghong <englefly@gmail.com>
morrySnow added a commit that referenced this pull request Jun 19, 2024
dataroaring pushed a commit that referenced this pull request Jun 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.0.0-merged meta-change reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants