Skip to content

[Pick 2.0](inverted index) fix wrong no need read data when need_remaining_after_evaluate #36685

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
xiaokang merged 4 commits into from
Jun 23, 2024
Merged

[Pick 2.0](inverted index) fix wrong no need read data when need_remaining_after_evaluate #36685

xiaokang merged 4 commits into from
Jun 23, 2024

Conversation

@airborne12
Copy link
Member

@airborne12 airborne12 commented Jun 21, 2024

When using an equal predicate on a column that applies an inverted index with a parser, it requires remaining_after_evaluate. In this situation, we cannot optimize the column without reading the data.

Proposed changes

Pick from (#36637)

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@airborne12
[Fix](inverted index) fix wrong no need read data when need_remaining…
e214409 
…_after_evaluate (apache#36637)

When using an equal predicate on a column that applies an inverted index
with a parser, it requires remaining_after_evaluate. In this situation,
we cannot optimize the column without reading the data.
@airborne12 airborne12 changed the title [Fix](inverted index) fix wrong no need read data when need_remaining_after_evaluate [Pick 2.0](inverted index) fix wrong no need read data when need_remaining_after_evaluate Jun 21, 2024
@airborne12
Copy link
Member Author

airborne12 commented Jun 21, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Jun 21, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

TPC-H: Total hot run time: 49618 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e2144096a373c579f5410b41909b19a1b1261870, data reload: false

------ Round 1 ----------------------------------
q1	17643	4382	4398	4382
q2	2075	155	157	155
q3	10412	1911	1959	1911
q4	10339	1262	1334	1262
q5	8418	3923	3911	3911
q6	234	147	125	125
q7	2031	1582	1588	1582
q8	9526	2732	2740	2732
q9	13714	10246	10154	10154
q10	8683	3495	3478	3478
q11	409	245	250	245
q12	469	308	306	306
q13	18330	3983	3983	3983
q14	353	320	319	319
q15	502	448	454	448
q16	673	572	569	569
q17	1120	944	933	933
q18	7356	6724	6832	6724
q19	1798	1618	1689	1618
q20	557	309	311	309
q21	4467	4131	4029	4029
q22	526	465	443	443
Total cold run time: 119635 ms
Total hot run time: 49618 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4298	4316	4306	4306
q2	322	226	219	219
q3	4160	4157	4113	4113
q4	2734	2734	2752	2734
q5	7128	7032	7074	7032
q6	238	122	120	120
q7	3221	2838	2847	2838
q8	4339	4468	4442	4442
q9	16954	16757	16675	16675
q10	4220	4296	4309	4296
q11	730	650	683	650
q12	1032	853	864	853
q13	7073	3724	3717	3717
q14	474	423	424	423
q15	504	462	453	453
q16	736	691	683	683
q17	3801	3810	3882	3810
q18	8712	8798	8715	8715
q19	1728	1650	1642	1642
q20	2364	2101	2128	2101
q21	8358	8405	8497	8405
q22	1064	992	954	954
Total cold run time: 84190 ms
Total hot run time: 79181 ms

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

TPC-DS: Total hot run time: 202825 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e2144096a373c579f5410b41909b19a1b1261870, data reload: false

query1	936	411	375	375
query2	6541	2852	2764	2764
query3	6928	201	196	196
query4	20008	17864	17890	17864
query5	19724	6461	6446	6446
query6	294	222	234	222
query7	4159	289	301	289
query8	442	446	430	430
query9	3085	2656	2561	2561
query10	419	293	300	293
query11	11343	10758	10752	10752
query12	126	73	75	73
query13	5596	707	666	666
query14	18045	13535	13348	13348
query15	367	238	242	238
query16	6471	275	260	260
query17	1715	1485	869	869
query18	2310	412	410	410
query19	209	152	148	148
query20	81	79	82	79
query21	188	105	92	92
query22	5131	4943	5000	4943
query23	32468	31938	31926	31926
query24	6904	6516	6526	6516
query25	543	418	428	418
query26	529	160	161	160
query27	1876	294	292	292
query28	6066	2349	2291	2291
query29	2815	2668	2802	2668
query30	247	162	167	162
query31	900	748	734	734
query32	73	64	60	60
query33	405	261	250	250
query34	846	472	476	472
query35	1114	888	943	888
query36	1403	1189	1258	1189
query37	89	62	59	59
query38	3049	2864	2960	2864
query39	1360	1346	1302	1302
query40	208	95	92	92
query41	45	43	42	42
query42	78	83	78	78
query43	714	708	707	707
query44	1118	729	726	726
query45	249	236	236	236
query46	1235	971	960	960
query47	1803	1746	1749	1746
query48	1011	710	692	692
query49	624	364	367	364
query50	859	621	611	611
query51	4785	4684	4735	4684
query52	91	82	95	82
query53	450	324	313	313
query54	2646	2433	2456	2433
query55	88	80	79	79
query56	237	219	209	209
query57	1244	1018	1203	1018
query58	203	212	196	196
query59	4231	4056	4112	4056
query60	209	216	213	213
query61	94	93	97	93
query62	764	447	431	431
query63	491	341	343	341
query64	4175	1649	1400	1400
query65	3620	3539	3579	3539
query66	668	374	375	374
query67	16199	15009	15513	15009
query68	9064	644	642	642
query69	574	335	343	335
query70	1615	1277	1291	1277
query71	415	301	315	301
query72	6503	3495	3475	3475
query73	742	319	314	314
query74	6323	5877	5878	5877
query75	5343	3690	3645	3645
query76	5430	1132	1227	1132
query77	914	252	250	250
query78	12554	11798	11798	11798
query79	9335	659	652	652
query80	1539	393	404	393
query81	485	230	242	230
query82	1641	100	95	95
query83	177	129	128	128
query84	258	71	73	71
query85	891	317	311	311
query86	348	282	293	282
query87	3188	2993	2997	2993
query88	4829	2285	2305	2285
query89	481	279	314	279
query90	1952	220	210	210
query91	179	143	139	139
query92	62	53	52	52
query93	6817	589	587	587
query94	711	215	207	207
query95	1121	1061	1075	1061
query96	647	322	319	319
query97	6431	6171	6402	6171
query98	189	171	166	166
query99	3039	870	855	855
Total cold run time: 316741 ms
Total hot run time: 202825 ms

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

TeamCity be ut coverage result:
Function Coverage: 37.87% (8109/21415)
Line Coverage: 29.53% (66361/224710)
Region Coverage: 29.00% (34203/117923)
Branch Coverage: 24.87% (17558/70600)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e2144096a373c579f5410b41909b19a1b1261870_e2144096a373c579f5410b41909b19a1b1261870/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

ClickBench: Total hot run time: 31.33 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e2144096a373c579f5410b41909b19a1b1261870, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.04	0.05
query4	1.80	0.07	0.07
query5	0.54	0.52	0.53
query6	1.32	0.63	0.61
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.53	0.49	0.48
query10	0.53	0.55	0.53
query11	0.11	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.62	0.61
query14	0.79	0.77	0.78
query15	0.77	0.76	0.76
query16	0.36	0.37	0.37
query17	1.01	1.02	1.02
query18	0.20	0.26	0.24
query19	1.86	1.88	1.81
query20	0.02	0.01	0.01
query21	15.46	0.57	0.56
query22	1.82	1.74	2.02
query23	17.16	1.05	1.15
query24	4.68	1.68	1.37
query25	0.37	0.10	0.06
query26	0.56	0.15	0.16
query27	0.04	0.04	0.03
query28	7.43	0.73	0.73
query29	12.70	2.33	2.28
query30	0.54	0.52	0.48
query31	2.81	0.40	0.36
query32	3.38	0.49	0.50
query33	3.08	3.08	3.07
query34	15.26	4.81	4.78
query35	4.86	4.84	4.84
query36	1.05	1.01	1.01
query37	0.06	0.04	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 102.57 s
Total hot run time: 31.33 s

@doris-robot
Copy link

doris-robot commented Jun 21, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e2144096a373c579f5410b41909b19a1b1261870 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

@xiaokang
Copy link
Contributor

xiaokang commented Jun 23, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-H: Total hot run time: 49876 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3b53f5efc681bc6301ae748c02710a7809d0edb4, data reload: false

------ Round 1 ----------------------------------
q1	17589	4406	4391	4391
q2	2071	158	160	158
q3	10401	1922	1939	1922
q4	10327	1259	1322	1259
q5	8506	3943	3915	3915
q6	233	125	125	125
q7	2033	1610	1599	1599
q8	9544	2721	2694	2694
q9	13586	10587	10424	10424
q10	8590	3499	3442	3442
q11	423	250	244	244
q12	470	303	308	303
q13	18332	3948	4042	3948
q14	355	330	335	330
q15	513	460	455	455
q16	661	572	572	572
q17	1128	931	932	931
q18	7203	6787	6838	6787
q19	1743	1568	1600	1568
q20	514	301	310	301
q21	4353	4066	4080	4066
q22	533	442	453	442
Total cold run time: 119108 ms
Total hot run time: 49876 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4316	4318	4302	4302
q2	318	223	215	215
q3	4181	4141	4132	4132
q4	2747	2711	2730	2711
q5	7164	7123	7094	7094
q6	237	121	121	121
q7	3247	2817	2817	2817
q8	4315	4470	4496	4470
q9	17484	17371	17084	17084
q10	4190	4219	4210	4210
q11	751	684	673	673
q12	1020	854	868	854
q13	6621	3752	3782	3752
q14	431	434	425	425
q15	507	464	460	460
q16	729	693	679	679
q17	3767	3747	3831	3747
q18	8733	8715	8777	8715
q19	1723	1688	1731	1688
q20	2399	2171	2117	2117
q21	8402	8487	8415	8415
q22	1017	1010	969	969
Total cold run time: 84299 ms
Total hot run time: 79650 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TeamCity be ut coverage result:
Function Coverage: 37.86% (8107/21415)
Line Coverage: 29.52% (66346/224726)
Region Coverage: 28.99% (34193/117934)
Branch Coverage: 24.86% (17552/70606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3b53f5efc681bc6301ae748c02710a7809d0edb4_3b53f5efc681bc6301ae748c02710a7809d0edb4/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-DS: Total hot run time: 203942 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3b53f5efc681bc6301ae748c02710a7809d0edb4, data reload: false

query1	939	420	377	377
query2	6532	2800	2725	2725
query3	6921	209	205	205
query4	21095	18058	18069	18058
query5	19727	6587	6554	6554
query6	343	213	222	213
query7	4586	307	318	307
query8	453	438	411	411
query9	3197	2758	2680	2680
query10	439	320	311	311
query11	11977	11402	11008	11008
query12	132	80	75	75
query13	5612	703	708	703
query14	20360	13654	13519	13519
query15	369	230	241	230
query16	6450	282	267	267
query17	1643	1454	876	876
query18	2419	429	415	415
query19	213	153	153	153
query20	80	76	81	76
query21	190	93	92	92
query22	5213	5064	4991	4991
query23	32504	31886	31833	31833
query24	6804	6591	6570	6570
query25	546	428	411	411
query26	489	165	161	161
query27	1711	297	298	297
query28	6147	2347	2294	2294
query29	2875	2871	2974	2871
query30	246	164	167	164
query31	912	726	759	726
query32	68	62	63	62
query33	408	251	273	251
query34	853	467	493	467
query35	1128	949	996	949
query36	1217	1137	1268	1137
query37	94	60	63	60
query38	3056	2904	2963	2904
query39	1370	1346	1318	1318
query40	207	89	94	89
query41	46	46	43	43
query42	86	88	84	84
query43	760	677	671	671
query44	1130	720	739	720
query45	254	236	236	236
query46	1233	980	975	975
query47	1961	1601	1710	1601
query48	985	712	705	705
query49	625	368	387	368
query50	858	642	623	623
query51	4764	4699	4626	4626
query52	93	76	76	76
query53	450	337	322	322
query54	2662	2485	2489	2485
query55	101	83	80	80
query56	244	224	216	216
query57	1164	1172	1076	1076
query58	218	190	217	190
query59	4054	3828	4095	3828
query60	229	206	215	206
query61	100	97	100	97
query62	785	441	552	441
query63	490	353	345	345
query64	2406	1558	1388	1388
query65	3668	3534	3556	3534
query66	773	374	366	366
query67	15705	14946	16989	14946
query68	10102	644	656	644
query69	592	362	335	335
query70	1927	1406	1300	1300
query71	417	311	307	307
query72	6518	3501	3557	3501
query73	738	317	319	317
query74	6242	5820	5926	5820
query75	5421	3637	3686	3637
query76	6391	1169	1231	1169
query77	1094	258	263	258
query78	12711	11911	11819	11819
query79	6336	625	628	625
query80	879	407	407	407
query81	483	242	232	232
query82	1006	102	101	101
query83	192	136	131	131
query84	262	75	70	70
query85	868	326	331	326
query86	328	332	290	290
query87	3188	3002	3034	3002
query88	4454	2313	2294	2294
query89	364	273	305	273
query90	1967	204	209	204
query91	184	134	139	134
query92	63	49	54	49
query93	5571	596	604	596
query94	716	213	218	213
query95	1116	1080	1053	1053
query96	635	329	319	319
query97	6539	6385	6379	6379
query98	197	171	175	171
query99	2828	908	936	908
Total cold run time: 315698 ms
Total hot run time: 203942 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

ClickBench: Total hot run time: 30.9 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3b53f5efc681bc6301ae748c02710a7809d0edb4, data reload: false

query1	0.02	0.02	0.02
query2	0.06	0.03	0.02
query3	0.24	0.04	0.04
query4	1.80	0.07	0.10
query5	0.54	0.53	0.52
query6	1.24	0.62	0.62
query7	0.01	0.01	0.02
query8	0.03	0.02	0.02
query9	0.53	0.49	0.49
query10	0.54	0.52	0.54
query11	0.12	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.62
query14	0.80	0.78	0.78
query15	0.78	0.76	0.75
query16	0.35	0.38	0.37
query17	1.02	0.98	1.03
query18	0.23	0.25	0.21
query19	1.93	1.87	1.81
query20	0.01	0.02	0.01
query21	15.52	0.56	0.54
query22	2.36	2.81	1.47
query23	17.39	1.07	0.90
query24	4.46	1.42	1.75
query25	0.30	0.16	0.05
query26	0.58	0.17	0.14
query27	0.03	0.04	0.03
query28	7.81	0.73	0.72
query29	12.70	2.21	2.31
query30	0.62	0.53	0.55
query31	2.81	0.38	0.37
query32	3.40	0.49	0.49
query33	3.08	3.07	3.08
query34	15.24	4.85	4.82
query35	4.87	4.88	4.90
query36	1.08	1.02	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.16	0.13	0.14
query41	0.06	0.01	0.01
query42	0.02	0.01	0.02
query43	0.03	0.01	0.02
Total cold run time: 103.62 s
Total hot run time: 30.9 s

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3b53f5efc681bc6301ae748c02710a7809d0edb4 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.0 seconds inserted 10000000 Rows, about 454K ops/s

@xiaokang
Copy link
Contributor

xiaokang commented Jun 23, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TeamCity be ut coverage result:
Function Coverage: 37.87% (8109/21415)
Line Coverage: 29.53% (66369/224726)
Region Coverage: 29.01% (34207/117934)
Branch Coverage: 24.87% (17559/70606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51_d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-H: Total hot run time: 49597 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51, data reload: false

------ Round 1 ----------------------------------
q1	17996	4400	4341	4341
q2	2081	153	147	147
q3	10468	1917	1926	1917
q4	10319	1270	1332	1270
q5	8533	3898	3945	3898
q6	237	128	126	126
q7	2050	1618	1577	1577
q8	9327	2758	2700	2700
q9	10695	10284	10231	10231
q10	8662	3474	3487	3474
q11	427	235	249	235
q12	470	304	307	304
q13	18370	3971	4035	3971
q14	362	329	332	329
q15	507	453	465	453
q16	681	572	567	567
q17	1120	960	973	960
q18	7234	6982	6765	6765
q19	1780	1671	1570	1570
q20	528	309	300	300
q21	4419	4118	4032	4032
q22	544	435	430	430
Total cold run time: 116810 ms
Total hot run time: 49597 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4280	4357	4312	4312
q2	322	230	223	223
q3	4172	4155	4122	4122
q4	2740	2733	2745	2733
q5	7133	7101	7095	7095
q6	240	122	118	118
q7	3233	2833	2857	2833
q8	4349	4456	4462	4456
q9	16936	16809	16826	16809
q10	4211	4253	4264	4253
q11	765	672	695	672
q12	1054	855	846	846
q13	6870	3746	3737	3737
q14	453	422	425	422
q15	515	457	456	456
q16	726	683	675	675
q17	3861	3854	3930	3854
q18	8929	8797	8844	8797
q19	1721	1723	1691	1691
q20	2341	2144	2078	2078
q21	8458	8430	8477	8430
q22	1055	982	971	971
Total cold run time: 84364 ms
Total hot run time: 79583 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-DS: Total hot run time: 203724 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51, data reload: false

query1	921	424	381	381
query2	6557	2839	2699	2699
query3	6913	205	199	199
query4	19884	17980	18003	17980
query5	19734	6479	6513	6479
query6	281	226	231	226
query7	4156	308	303	303
query8	429	425	393	393
query9	3094	2669	2609	2609
query10	415	303	295	295
query11	11247	10744	10674	10674
query12	120	79	84	79
query13	5619	685	683	683
query14	18333	13665	13687	13665
query15	362	250	250	250
query16	6478	286	262	262
query17	1753	1457	871	871
query18	2324	414	416	414
query19	204	155	152	152
query20	75	78	81	78
query21	204	100	96	96
query22	5199	4975	4921	4921
query23	32821	32186	31966	31966
query24	6926	6544	6556	6544
query25	535	440	428	428
query26	531	164	165	164
query27	1859	299	298	298
query28	6110	2365	2309	2309
query29	2834	2801	2686	2686
query30	237	165	168	165
query31	923	739	753	739
query32	73	62	66	62
query33	414	255	248	248
query34	852	473	488	473
query35	1108	940	978	940
query36	1356	1040	1192	1040
query37	91	64	60	60
query38	3069	2927	2887	2887
query39	1391	1354	1335	1335
query40	207	89	92	89
query41	46	43	43	43
query42	83	87	81	81
query43	706	666	696	666
query44	1149	716	729	716
query45	249	253	236	236
query46	1237	964	964	964
query47	1729	1636	1709	1636
query48	1014	729	700	700
query49	624	375	383	375
query50	863	589	640	589
query51	4764	4663	4707	4663
query52	88	91	77	77
query53	455	320	316	316
query54	2705	2486	2488	2486
query55	99	94	88	88
query56	245	207	223	207
query57	1124	1089	1095	1089
query58	222	197	199	197
query59	4160	4257	3911	3911
query60	219	217	223	217
query61	100	95	97	95
query62	888	514	456	456
query63	491	346	348	346
query64	2337	1563	1432	1432
query65	3628	3544	4025	3544
query66	811	387	376	376
query67	15918	15615	17432	15615
query68	8633	640	654	640
query69	567	355	356	355
query70	1739	1577	1371	1371
query71	423	309	311	309
query72	6529	3511	3561	3511
query73	719	314	321	314
query74	6253	5828	5878	5828
query75	5164	3751	3676	3676
query76	5124	1144	1188	1144
query77	740	250	266	250
query78	12356	11648	11514	11514
query79	6769	640	636	636
query80	939	403	405	403
query81	490	244	236	236
query82	1520	102	102	102
query83	181	138	134	134
query84	263	72	72	72
query85	882	327	324	324
query86	332	293	293	293
query87	3182	3033	3022	3022
query88	4884	2299	2292	2292
query89	386	283	294	283
query90	1958	210	208	208
query91	193	138	140	138
query92	62	52	54	52
query93	5041	550	567	550
query94	721	206	219	206
query95	1111	1061	1045	1045
query96	643	321	341	321
query97	6496	6259	6447	6259
query98	183	175	163	163
query99	2986	810	931	810
Total cold run time: 309062 ms
Total hot run time: 203724 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

ClickBench: Total hot run time: 30.83 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.02
query3	0.25	0.05	0.05
query4	1.80	0.07	0.06
query5	0.55	0.53	0.51
query6	1.24	0.62	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.49	0.48
query10	0.55	0.54	0.53
query11	0.12	0.09	0.09
query12	0.11	0.10	0.09
query13	0.62	0.61	0.61
query14	0.78	0.79	0.79
query15	0.80	0.77	0.77
query16	0.39	0.37	0.38
query17	0.99	1.03	0.97
query18	0.21	0.28	0.21
query19	1.91	1.86	1.82
query20	0.01	0.02	0.01
query21	15.48	0.54	0.54
query22	1.95	2.87	1.99
query23	17.19	1.13	0.92
query24	6.43	0.87	1.06
query25	0.34	0.12	0.06
query26	0.65	0.17	0.17
query27	0.05	0.04	0.04
query28	6.91	0.79	0.73
query29	12.72	2.18	2.26
query30	0.60	0.53	0.50
query31	2.81	0.39	0.38
query32	3.39	0.50	0.48
query33	3.05	3.06	3.08
query34	15.25	4.81	4.78
query35	4.84	4.84	4.84
query36	1.04	1.02	1.00
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.02	0.02
query40	0.16	0.14	0.14
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.02	0.01	0.01
Total cold run time: 104.07 s
Total hot run time: 30.83 s

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d0dcd10fd1fb2875eecfc273bbaaca3f6d1b0f51 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.1 seconds inserted 10000000 Rows, about 473K ops/s

@xiaokang
Copy link
Contributor

xiaokang commented Jun 23, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Jun 23, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-H: Total hot run time: 49547 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e093b80297e47f88a06b882aeed1597863a3914d, data reload: false

------ Round 1 ----------------------------------
q1	18164	4351	4328	4328
q2	2039	159	144	144
q3	10441	1858	1931	1858
q4	10330	1216	1285	1216
q5	8748	3950	3920	3920
q6	235	125	126	125
q7	2076	1609	1586	1586
q8	9262	2706	2700	2700
q9	10577	10361	10244	10244
q10	8634	3506	3459	3459
q11	423	246	239	239
q12	473	307	308	307
q13	18347	3951	3995	3951
q14	348	335	328	328
q15	509	461	467	461
q16	664	570	561	561
q17	1133	979	977	977
q18	7334	6849	6748	6748
q19	1809	1637	1626	1626
q20	553	297	290	290
q21	4408	4094	4031	4031
q22	535	456	448	448
Total cold run time: 117042 ms
Total hot run time: 49547 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4265	4279	4265
q2	321	227	218	218
q3	4188	4139	4133	4133
q4	2755	2743	2742	2742
q5	7195	7073	7113	7073
q6	235	122	120	120
q7	3261	2892	2852	2852
q8	4371	4439	4497	4439
q9	16914	16800	16844	16800
q10	4244	4257	4300	4257
q11	730	703	715	703
q12	1045	864	864	864
q13	7047	3754	3736	3736
q14	457	435	422	422
q15	509	454	456	454
q16	739	670	674	670
q17	3844	3832	3841	3832
q18	8931	8817	8688	8688
q19	1698	1719	1631	1631
q20	2374	2112	2211	2112
q21	8450	8464	8418	8418
q22	1057	982	986	982
Total cold run time: 84703 ms
Total hot run time: 79411 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TeamCity be ut coverage result:
Function Coverage: 37.86% (8107/21415)
Line Coverage: 29.52% (66348/224726)
Region Coverage: 28.99% (34192/117934)
Branch Coverage: 24.86% (17553/70606)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e093b80297e47f88a06b882aeed1597863a3914d_e093b80297e47f88a06b882aeed1597863a3914d/report/index.html

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

TPC-DS: Total hot run time: 203372 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e093b80297e47f88a06b882aeed1597863a3914d, data reload: false

query1	930	394	407	394
query2	6562	2871	2510	2510
query3	6926	205	203	203
query4	20967	18028	18113	18028
query5	19726	6513	6493	6493
query6	281	226	237	226
query7	4159	304	307	304
query8	401	403	454	403
query9	3091	2662	2598	2598
query10	408	324	310	310
query11	11303	10686	10706	10686
query12	130	75	71	71
query13	5606	689	686	686
query14	17865	13191	13971	13191
query15	368	251	241	241
query16	6468	297	268	268
query17	1696	1451	889	889
query18	2298	434	416	416
query19	205	160	151	151
query20	79	81	80	80
query21	189	95	93	93
query22	5119	5037	5034	5034
query23	32590	31950	31961	31950
query24	7016	6539	6538	6538
query25	538	426	439	426
query26	526	164	169	164
query27	1878	297	299	297
query28	6149	2337	2293	2293
query29	2799	2781	2695	2695
query30	246	169	166	166
query31	921	729	730	729
query32	72	65	61	61
query33	397	276	254	254
query34	877	483	463	463
query35	1132	926	851	851
query36	1454	1147	1100	1100
query37	93	65	61	61
query38	3096	2929	2956	2929
query39	1379	1344	1320	1320
query40	214	90	103	90
query41	47	44	46	44
query42	86	86	93	86
query43	745	740	709	709
query44	1149	730	726	726
query45	250	238	238	238
query46	1217	988	983	983
query47	1845	1717	1708	1708
query48	1016	705	710	705
query49	624	369	369	369
query50	877	624	605	605
query51	4784	4649	4654	4649
query52	98	84	75	75
query53	456	320	322	320
query54	2687	2479	2475	2475
query55	92	83	78	78
query56	241	216	224	216
query57	1333	1107	1184	1107
query58	220	210	191	191
query59	4277	3901	4012	3901
query60	210	195	205	195
query61	101	98	98	98
query62	826	472	477	472
query63	482	354	345	345
query64	2522	1568	1506	1506
query65	3663	3549	3591	3549
query66	752	388	381	381
query67	18242	15917	15187	15187
query68	9285	672	644	644
query69	584	350	344	344
query70	1681	1416	1578	1416
query71	429	313	321	313
query72	6549	3459	3511	3459
query73	741	314	322	314
query74	6354	5879	5877	5877
query75	5359	3711	3667	3667
query76	5813	1148	1220	1148
query77	968	256	255	255
query78	12486	11649	12040	11649
query79	8066	637	626	626
query80	1195	406	410	406
query81	485	238	233	233
query82	1711	103	101	101
query83	181	138	138	138
query84	255	74	74	74
query85	886	325	323	323
query86	342	295	310	295
query87	3222	3073	3029	3029
query88	4968	2290	2295	2290
query89	385	292	296	292
query90	1968	202	210	202
query91	171	135	142	135
query92	60	58	56	56
query93	5910	600	602	600
query94	700	227	212	212
query95	1101	1062	1080	1062
query96	628	328	333	328
query97	6467	6396	6292	6292
query98	188	178	172	172
query99	2899	840	953	840
Total cold run time: 317033 ms
Total hot run time: 203372 ms

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

ClickBench: Total hot run time: 31.23 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e093b80297e47f88a06b882aeed1597863a3914d, data reload: false

query1	0.02	0.02	0.03
query2	0.07	0.02	0.02
query3	0.24	0.05	0.06
query4	1.78	0.07	0.07
query5	0.54	0.53	0.52
query6	1.24	0.64	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.54	0.49	0.48
query10	0.54	0.54	0.53
query11	0.12	0.09	0.08
query12	0.12	0.10	0.09
query13	0.62	0.62	0.61
query14	0.80	0.79	0.79
query15	0.79	0.76	0.77
query16	0.36	0.37	0.36
query17	1.02	1.01	1.00
query18	0.23	0.26	0.25
query19	1.94	1.77	1.88
query20	0.02	0.01	0.01
query21	15.46	0.55	0.54
query22	2.36	2.34	1.64
query23	17.36	1.10	0.97
query24	5.61	3.73	1.36
query25	0.38	0.08	0.06
query26	0.85	0.15	0.15
query27	0.04	0.04	0.03
query28	4.70	0.78	0.77
query29	12.64	2.28	2.30
query30	0.56	0.54	0.62
query31	2.82	0.39	0.37
query32	3.37	0.50	0.49
query33	3.06	3.06	3.04
query34	15.28	4.79	4.80
query35	4.83	4.87	4.85
query36	1.06	1.02	1.02
query37	0.06	0.05	0.04
query38	0.03	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.15	0.14
query41	0.06	0.01	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 101.8 s
Total hot run time: 31.23 s

@doris-robot
Copy link

doris-robot commented Jun 23, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit e093b80297e47f88a06b882aeed1597863a3914d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@xiaokang xiaokang merged commit b35b792 into apache:branch-2.0 Jun 23, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@airborne12
[Pick 2.0](inverted index) fix wrong no need read data when need_rema…
fe08a9a 
…ining_after_evaluate apache#36637 (apache#36685)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

kind/test

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@airborne12 @doris-robot @xiaokang