Skip to content

[Fix](group commit) Fix pre allocated err handling for group commit async load and add regression test#30718

Merged
yiguolei merged 1 commit intoapache:masterfrom
Yukang-Lian:Fix_Pre_Allocated_Err_Handling_For_Group_Commit_Async_Load
Feb 1, 2024
Merged

[Fix](group commit) Fix pre allocated err handling for group commit async load and add regression test#30718
yiguolei merged 1 commit intoapache:masterfrom
Yukang-Lian:Fix_Pre_Allocated_Err_Handling_For_Group_Commit_Async_Load

Conversation

@Yukang-Lian
Copy link
Collaborator

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@Yukang-Lian
Copy link
Collaborator Author

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Feb 1, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37005 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 54fda652a38c283ae225f09f90c9d66edec1455e, data reload: false

------ Round 1 ----------------------------------
q1	17658	4523	4393	4393
q2	2044	142	134	134
q3	10589	933	952	933
q4	4650	781	734	734
q5	7666	2858	2782	2782
q6	185	122	120	120
q7	1139	718	725	718
q8	9308	2012	2009	2009
q9	7288	6355	6318	6318
q10	8098	2424	2448	2424
q11	414	212	216	212
q12	744	283	264	264
q13	18031	3312	3319	3312
q14	278	255	244	244
q15	531	503	491	491
q16	484	408	406	406
q17	952	565	532	532
q18	6803	6034	5942	5942
q19	1567	1412	1358	1358
q20	575	352	348	348
q21	6860	3079	3021	3021
q22	811	310	310	310
Total cold run time: 106675 ms
Total hot run time: 37005 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4503	4371	4410	4371
q2	335	231	241	231
q3	2967	2857	2824	2824
q4	1857	1604	1642	1604
q5	5192	5245	5248	5245
q6	187	113	115	113
q7	2173	1765	1813	1765
q8	3093	3237	3253	3237
q9	8420	8367	8316	8316
q10	5902	3570	3582	3570
q11	545	465	476	465
q12	731	588	592	588
q13	11739	3145	3085	3085
q14	286	240	262	240
q15	527	494	484	484
q16	529	480	477	477
q17	1829	1706	1697	1697
q18	7953	7707	7522	7522
q19	8681	1561	1545	1545
q20	2140	1919	1901	1901
q21	4895	4642	4504	4504
q22	545	468	451	451
Total cold run time: 75029 ms
Total hot run time: 54235 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 176034 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 54fda652a38c283ae225f09f90c9d66edec1455e, data reload: false

query1	940	342	340	340
query2	6547	1940	1966	1940
query3	6704	201	201	201
query4	30989	22067	22102	22067
query5	4463	433	351	351
query6	251	156	154	154
query7	4610	266	273	266
query8	247	173	179	173
query9	8911	2299	2294	2294
query10	405	205	201	201
query11	18808	15488	15403	15403
query12	121	68	64	64
query13	1616	385	388	385
query14	10014	7633	7452	7452
query15	231	183	186	183
query16	7347	278	248	248
query17	1764	494	465	465
query18	1938	264	253	253
query19	181	135	136	135
query20	74	74	71	71
query21	198	135	135	135
query22	4938	4681	4691	4681
query23	31043	30441	30308	30308
query24	6416	2757	2781	2757
query25	484	316	311	311
query26	724	147	140	140
query27	2002	283	281	281
query28	3603	1927	1912	1912
query29	917	613	606	606
query30	224	134	135	134
query31	919	701	716	701
query32	94	50	50	50
query33	421	210	203	203
query34	832	455	473	455
query35	862	763	757	757
query36	1333	1197	1166	1166
query37	92	55	61	55
query38	3357	3185	3179	3179
query39	1314	1254	1259	1254
query40	198	87	85	85
query41	37	35	36	35
query42	87	83	81	81
query43	541	499	502	499
query44	1059	708	710	708
query45	191	185	177	177
query46	1026	638	647	638
query47	1602	1470	1506	1470
query48	404	308	309	308
query49	648	280	284	280
query50	695	311	318	311
query51	5292	5199	5132	5132
query52	99	76	83	76
query53	324	266	275	266
query54	225	184	192	184
query55	79	76	74	74
query56	180	160	162	160
query57	957	909	890	890
query58	188	149	158	149
query59	2487	2279	2336	2279
query60	201	178	185	178
query61	84	87	84	84
query62	514	358	356	356
query63	287	270	274	270
query64	4275	3346	3812	3346
query65	3264	3253	3245	3245
query66	945	320	309	309
query67	14470	14229	14305	14229
query68	4221	530	519	519
query69	451	319	305	305
query70	1547	1518	1530	1518
query71	302	218	207	207
query72	5550	3125	2872	2872
query73	697	319	324	319
query74	6800	6219	6227	6219
query75	2895	2349	2282	2282
query76	2490	986	990	986
query77	395	234	231	231
query78	9147	8832	8642	8642
query79	2586	498	487	487
query80	2119	332	325	325
query81	528	204	211	204
query82	710	88	85	85
query83	248	117	118	117
query84	334	78	79	78
query85	2025	350	335	335
query86	518	371	386	371
query87	3415	3232	3256	3232
query88	3745	2171	2170	2170
query89	431	355	348	348
query90	2019	183	183	183
query91	168	118	119	118
query92	52	43	45	43
query93	2913	455	423	423
query94	1355	159	156	156
query95	500	450	447	447
query96	635	317	316	316
query97	4243	4093	4112	4093
query98	211	195	195	195
query99	1150	686	658	658
Total cold run time: 270693 ms
Total hot run time: 176034 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 36.17% (8627/23849)
Line Coverage: 28.23% (70516/249781)
Region Coverage: 27.23% (36394/133637)
Branch Coverage: 24.02% (18635/77592)
Coverage Report: http://coverage.selectdb-in.cc/coverage/54fda652a38c283ae225f09f90c9d66edec1455e_54fda652a38c283ae225f09f90c9d66edec1455e/report/index.html

@doris-robot
Copy link

ClickBench: Total hot run time: 29.89 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 54fda652a38c283ae225f09f90c9d66edec1455e, data reload: false

query1	0.04	0.04	0.03
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.67	0.09	0.10
query5	0.53	0.53	0.51
query6	1.23	0.65	0.66
query7	0.02	0.01	0.01
query8	0.04	0.02	0.02
query9	0.55	0.49	0.49
query10	0.55	0.55	0.56
query11	0.12	0.09	0.09
query12	0.11	0.08	0.09
query13	0.60	0.62	0.61
query14	0.79	0.80	0.81
query15	0.80	0.77	0.78
query16	0.37	0.39	0.39
query17	1.04	1.01	1.02
query18	0.24	0.25	0.26
query19	1.90	1.76	1.76
query20	0.02	0.01	0.01
query21	15.40	0.55	0.56
query22	2.52	3.03	1.26
query23	17.24	0.97	0.69
query24	2.50	1.15	0.38
query25	0.37	0.23	0.18
query26	0.37	0.13	0.14
query27	0.05	0.05	0.04
query28	12.14	0.84	0.87
query29	12.57	3.19	3.27
query30	0.65	0.51	0.55
query31	2.83	0.33	0.36
query32	3.36	0.49	0.48
query33	3.23	3.17	3.26
query34	15.88	4.36	4.34
query35	4.36	4.26	4.31
query36	1.11	1.07	1.06
query37	0.06	0.06	0.05
query38	0.04	0.03	0.03
query39	0.02	0.02	0.02
query40	0.16	0.15	0.13
query41	0.07	0.02	0.02
query42	0.03	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 105.89 s
Total hot run time: 29.89 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 54fda652a38c283ae225f09f90c9d66edec1455e with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

@github-actions
Copy link
Contributor

github-actions bot commented Feb 1, 2024

PR approved by anyone and no changes requested.

@yiguolei yiguolei merged commit 4b7f03b into apache:master Feb 1, 2024
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 1, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 1, 2024

PR approved by at least one committer and no changes requested.

yiguolei pushed a commit that referenced this pull request Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants