Skip to content

[improvement](inverted index)Avoid parsing ignore_above in add_values and return error status when rowid_conversion is failed. (#32890)#36200

Merged
xiaokang merged 2 commits intoapache:branch-2.0from
qidaye:pick_32809
Jun 16, 2024
Merged

[improvement](inverted index)Avoid parsing ignore_above in add_values and return error status when rowid_conversion is failed. (#32890)#36200
xiaokang merged 2 commits intoapache:branch-2.0from
qidaye:pick_32809

Conversation

@qidaye
Copy link
Contributor

@qidaye qidaye commented Jun 12, 2024

… and return error status when rowid_conversion is failed. (#32890)

pick from master #32890

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@qidaye
Copy link
Contributor Author

qidaye commented Jun 12, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8101/21424)
Line Coverage: 29.47% (66162/224484)
Region Coverage: 28.97% (34115/117778)
Branch Coverage: 24.81% (17508/70556)
Coverage Report: http://coverage.selectdb-in.cc/coverage/e6337eabda4a043e9d781fa3cb0c4be602ce7cb2_e6337eabda4a043e9d781fa3cb0c4be602ce7cb2/report/index.html

qidaye and others added 2 commits June 13, 2024 10:50
… and return error status when rowid_conversion is failed. (apache#32890)

Co-authored-by: Luennng <luennng@gmail.com>
@qidaye
Copy link
Contributor Author

qidaye commented Jun 13, 2024

run buildall

@github-actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 49688 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 840987d43db27cad8b66f8b42ad97e171c29efa4, data reload: false

------ Round 1 ----------------------------------
q1	17796	4354	4353	4353
q2	2062	153	151	151
q3	10477	1897	1937	1897
q4	10353	1277	1334	1277
q5	8775	3915	3904	3904
q6	254	128	122	122
q7	2073	1611	1623	1611
q8	9513	2741	2693	2693
q9	13369	10276	10232	10232
q10	8646	3495	3530	3495
q11	416	252	252	252
q12	470	313	307	307
q13	18349	3903	4058	3903
q14	348	335	329	329
q15	500	455	473	455
q16	689	591	574	574
q17	1134	933	957	933
q18	7245	6911	6777	6777
q19	1777	1655	1592	1592
q20	531	320	317	317
q21	4467	4097	4064	4064
q22	545	454	450	450
Total cold run time: 119789 ms
Total hot run time: 49688 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4320	4367	4285	4285
q2	316	227	226	226
q3	4145	4150	4101	4101
q4	2757	2749	2757	2749
q5	7107	7112	7062	7062
q6	242	121	122	121
q7	3203	2887	2788	2788
q8	4398	4463	4471	4463
q9	16836	16740	16686	16686
q10	4234	4266	4265	4265
q11	794	701	688	688
q12	1047	852	877	852
q13	7245	3739	3744	3739
q14	450	413	422	413
q15	517	458	457	457
q16	735	689	675	675
q17	3861	3768	3826	3768
q18	8686	8857	8904	8857
q19	1740	1641	1704	1641
q20	2362	2115	2111	2111
q21	8464	8386	8366	8366
q22	1112	984	991	984
Total cold run time: 84571 ms
Total hot run time: 79297 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.80% (8094/21411)
Line Coverage: 29.47% (66132/224382)
Region Coverage: 28.94% (34083/117780)
Branch Coverage: 24.81% (17494/70522)
Coverage Report: http://coverage.selectdb-in.cc/coverage/840987d43db27cad8b66f8b42ad97e171c29efa4_840987d43db27cad8b66f8b42ad97e171c29efa4/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 204449 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 840987d43db27cad8b66f8b42ad97e171c29efa4, data reload: false

query1	938	425	378	378
query2	6529	2801	2661	2661
query3	6920	216	206	206
query4	20104	18115	18161	18115
query5	19724	6497	6548	6497
query6	303	219	237	219
query7	4342	310	305	305
query8	412	428	396	396
query9	3134	2694	2641	2641
query10	424	314	299	299
query11	11374	10778	10770	10770
query12	127	81	71	71
query13	5603	707	689	689
query14	18530	13632	13669	13632
query15	361	246	265	246
query16	6446	291	265	265
query17	1405	1567	895	895
query18	2249	419	410	410
query19	210	149	155	149
query20	82	80	80	80
query21	191	99	95	95
query22	5491	5137	5147	5137
query23	32380	31979	31927	31927
query24	6781	6513	6503	6503
query25	535	439	428	428
query26	632	161	164	161
query27	2030	295	295	295
query28	6096	2406	2327	2327
query29	2863	2764	2677	2677
query30	247	163	171	163
query31	902	733	777	733
query32	64	66	62	62
query33	383	266	265	265
query34	857	476	494	476
query35	1136	907	920	907
query36	1308	1386	1293	1293
query37	93	63	63	63
query38	3113	2945	2936	2936
query39	1367	1340	1338	1338
query40	244	93	92	92
query41	46	45	46	45
query42	85	85	85	85
query43	790	659	678	659
query44	1178	738	719	719
query45	255	237	236	236
query46	1227	994	1030	994
query47	1693	1688	1647	1647
query48	1019	722	747	722
query49	628	373	382	373
query50	872	612	618	612
query51	4742	4668	4708	4668
query52	96	84	102	84
query53	458	338	331	331
query54	2667	2479	2512	2479
query55	82	81	85	81
query56	238	216	202	202
query57	1193	1071	1092	1071
query58	226	216	206	206
query59	4478	3917	3811	3811
query60	209	206	213	206
query61	99	95	100	95
query62	824	492	450	450
query63	496	348	358	348
query64	2532	1531	1489	1489
query65	3627	3549	3587	3549
query66	803	366	385	366
query67	15581	15559	17092	15559
query68	9916	660	660	660
query69	589	341	349	341
query70	1877	1486	1364	1364
query71	426	320	306	306
query72	6607	3475	3541	3475
query73	750	328	323	323
query74	6450	5909	5821	5821
query75	5362	3664	3728	3664
query76	6277	1172	1155	1155
query77	1030	263	254	254
query78	12381	11586	12792	11586
query79	10899	626	655	626
query80	1137	395	397	395
query81	461	236	235	235
query82	523	99	96	96
query83	178	130	135	130
query84	250	72	72	72
query85	911	322	322	322
query86	322	289	339	289
query87	3262	3006	3064	3006
query88	4911	2298	2280	2280
query89	376	274	324	274
query90	2596	191	211	191
query91	189	141	140	140
query92	57	51	51	51
query93	4872	569	541	541
query94	1072	204	211	204
query95	1121	1051	1034	1034
query96	642	326	316	316
query97	6496	6335	6386	6335
query98	194	185	174	174
query99	3223	923	900	900
Total cold run time: 317431 ms
Total hot run time: 204449 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.93 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 840987d43db27cad8b66f8b42ad97e171c29efa4, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.03	0.03
query3	0.26	0.05	0.04
query4	1.78	0.07	0.07
query5	0.54	0.52	0.52
query6	1.23	0.63	0.61
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.50
query10	0.53	0.53	0.54
query11	0.13	0.09	0.08
query12	0.12	0.08	0.08
query13	0.63	0.61	0.61
query14	0.81	0.78	0.78
query15	0.78	0.76	0.77
query16	0.35	0.36	0.37
query17	1.06	1.05	1.01
query18	0.21	0.26	0.26
query19	1.94	1.86	1.77
query20	0.01	0.01	0.02
query21	15.46	0.56	0.55
query22	2.22	2.34	1.72
query23	16.50	1.16	0.92
query24	7.54	1.23	1.14
query25	0.42	0.08	0.06
query26	0.78	0.16	0.15
query27	0.04	0.04	0.04
query28	5.27	0.82	0.78
query29	12.76	2.33	2.25
query30	0.61	0.53	0.53
query31	2.81	0.39	0.37
query32	3.36	0.50	0.51
query33	3.11	3.02	3.05
query34	15.28	4.81	4.77
query35	4.86	4.87	4.80
query36	1.05	1.02	1.02
query37	0.06	0.05	0.05
query38	0.03	0.03	0.02
query39	0.02	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.01	0.02
query42	0.02	0.01	0.02
query43	0.03	0.02	0.02
Total cold run time: 103.51 s
Total hot run time: 30.93 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 840987d43db27cad8b66f8b42ad97e171c29efa4 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@xiaokang xiaokang changed the title [improvement](inverted index)Avoid parsing ignore_above in add_values… [improvement](inverted index)Avoid parsing ignore_above in add_values and return error status when rowid_conversion is failed. (#32890) Jun 16, 2024
@xiaokang xiaokang merged commit e3fb361 into apache:branch-2.0 Jun 16, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
… and return error status when rowid_conversion is failed. (apache#32890) (apache#36200)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants