Skip to content

Conversation

@mymeiyi
Copy link
Contributor

@mymeiyi mymeiyi commented Mar 12, 2024

Proposed changes

Issue Number: close #xxx

When use group commmit and chunked stream load, because chunked stream load does not has content length, we can estimate the wal size, so we skip group commit in the original way.
This pr allow it.
How to avoid large wal will be solved in later pr.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Mar 12, 2024

run buildall

@mymeiyi
Copy link
Contributor Author

mymeiyi commented Mar 12, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 34.95% (8553/24470)
Line Coverage: 26.79% (69506/259419)
Region Coverage: 26.04% (36064/138507)
Branch Coverage: 23.01% (18438/80128)
Coverage Report: http://coverage.selectdb-in.cc/coverage/3bd3fbbef0cc708a3446e75e6e3b345a75ef5869_3bd3fbbef0cc708a3446e75e6e3b345a75ef5869/report/index.html

@doris-robot
Copy link

TPC-H: Total hot run time: 36651 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false

------ Round 1 ----------------------------------
q1	17675	4142	4072	4072
q2	2021	154	142	142
q3	10568	905	897	897
q4	4650	675	737	675
q5	7595	2800	2826	2800
q6	184	125	126	125
q7	1221	822	808	808
q8	9476	2061	2015	2015
q9	7305	6407	6438	6407
q10	8195	2523	2456	2456
q11	425	229	224	224
q12	763	309	297	297
q13	17952	3034	3012	3012
q14	278	262	249	249
q15	496	458	451	451
q16	485	405	396	396
q17	960	516	561	516
q18	6636	5873	5778	5778
q19	1549	1451	1419	1419
q20	554	286	272	272
q21	7494	3333	3354	3333
q22	793	307	339	307
Total cold run time: 107275 ms
Total hot run time: 36651 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4086	4097	4050	4050
q2	319	225	220	220
q3	2922	2816	2833	2816
q4	1824	1575	1563	1563
q5	5204	5278	5263	5263
q6	195	116	124	116
q7	2239	1831	1815	1815
q8	3146	3323	3286	3286
q9	8518	8499	8511	8499
q10	6158	3686	3678	3678
q11	533	460	459	459
q12	721	536	554	536
q13	15648	2864	2830	2830
q14	264	257	271	257
q15	502	444	444	444
q16	469	401	412	401
q17	1718	1480	1443	1443
q18	7554	7230	7118	7118
q19	6163	1539	1555	1539
q20	1911	1736	1725	1725
q21	4872	4553	4619	4553
q22	507	456	448	448
Total cold run time: 75473 ms
Total hot run time: 53059 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186755 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false

query1	906	356	347	347
query2	7422	2140	2023	2023
query3	6712	225	212	212
query4	31201	20612	20647	20612
query5	4338	400	420	400
query6	257	176	186	176
query7	4629	290	281	281
query8	229	174	173	173
query9	9881	2245	2214	2214
query10	455	235	262	235
query11	14542	14165	14253	14165
query12	130	85	79	79
query13	1615	407	402	402
query14	11178	10842	10360	10360
query15	271	221	189	189
query16	8106	270	272	270
query17	1934	568	540	540
query18	2105	285	278	278
query19	294	162	153	153
query20	92	86	88	86
query21	197	132	127	127
query22	4696	4390	4439	4390
query23	31809	31225	31099	31099
query24	10744	2823	2790	2790
query25	611	377	382	377
query26	1274	158	167	158
query27	2429	360	355	355
query28	7626	1855	1865	1855
query29	871	643	632	632
query30	301	146	146	146
query31	958	734	723	723
query32	96	61	58	58
query33	764	281	266	266
query34	1028	472	501	472
query35	831	619	607	607
query36	964	880	851	851
query37	122	77	78	77
query38	3326	3113	3201	3113
query39	1428	1377	1386	1377
query40	217	114	111	111
query41	51	48	46	46
query42	103	95	101	95
query43	497	467	467	467
query44	1118	730	697	697
query45	272	243	271	243
query46	1092	702	685	685
query47	1692	1608	1597	1597
query48	447	372	352	352
query49	1141	333	337	333
query50	762	383	384	383
query51	6635	6646	6486	6486
query52	109	91	94	91
query53	341	280	318	280
query54	307	250	242	242
query55	83	81	76	76
query56	261	239	225	225
query57	1077	1027	1007	1007
query58	240	199	205	199
query59	2743	2693	2671	2671
query60	267	273	254	254
query61	108	108	107	107
query62	600	408	398	398
query63	307	285	282	282
query64	5743	3765	3885	3765
query65	3101	3045	3020	3020
query66	815	327	327	327
query67	15040	14543	14800	14543
query68	6770	518	521	518
query69	619	378	377	377
query70	1289	1134	1172	1134
query71	486	270	263	263
query72	6509	2650	2479	2479
query73	733	322	319	319
query74	6989	6656	6533	6533
query75	9432	7980	7999	7980
query76	4356	839	904	839
query77	605	262	256	256
query78	10359	9634	9622	9622
query79	8699	529	527	527
query80	1635	376	373	373
query81	521	208	209	208
query82	879	204	202	202
query83	220	146	145	145
query84	281	75	75	75
query85	1471	322	310	310
query86	472	295	303	295
query87	3542	3231	3305	3231
query88	4971	2277	2271	2271
query89	500	352	363	352
query90	2001	181	173	173
query91	159	132	128	128
query92	63	45	46	45
query93	6664	501	496	496
query94	1112	178	173	173
query95	424	319	328	319
query96	614	278	270	270
query97	4440	4240	4321	4240
query98	226	216	207	207
query99	1117	762	745	745
Total cold run time: 309699 ms
Total hot run time: 186755 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.57 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869, data reload: false

query1	0.04	0.04	0.04
query2	0.06	0.03	0.02
query3	0.22	0.06	0.06
query4	1.67	0.10	0.09
query5	0.50	0.48	0.52
query6	1.18	0.63	0.63
query7	0.02	0.02	0.01
query8	0.04	0.03	0.03
query9	0.55	0.50	0.52
query10	0.55	0.54	0.56
query11	0.12	0.09	0.09
query12	0.11	0.09	0.09
query13	0.57	0.58	0.57
query14	0.74	0.74	0.75
query15	0.79	0.78	0.79
query16	0.41	0.39	0.39
query17	0.99	0.97	0.97
query18	0.19	0.27	0.21
query19	1.74	1.64	1.66
query20	0.01	0.01	0.01
query21	15.42	0.60	0.58
query22	2.65	2.58	1.27
query23	17.16	0.92	0.71
query24	4.32	1.59	0.72
query25	0.08	0.13	0.04
query26	0.85	0.12	0.11
query27	0.03	0.03	0.05
query28	10.35	0.92	0.87
query29	12.52	3.36	3.45
query30	0.67	0.63	0.56
query31	2.79	0.32	0.33
query32	3.41	0.44	0.44
query33	2.78	2.77	2.78
query34	16.10	4.31	4.31
query35	4.29	4.35	4.29
query36	1.08	1.00	1.00
query37	0.07	0.05	0.06
query38	0.05	0.03	0.03
query39	0.03	0.02	0.02
query40	0.17	0.15	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.02
Total cold run time: 105.46 s
Total hot run time: 29.57 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3bd3fbbef0cc708a3446e75e6e3b345a75ef5869 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       17.4 seconds inserted 10000000 Rows, about 574K ops/s

Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 13, 2024
@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Collaborator

@Yukang-Lian Yukang-Lian left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit b989ea1 into apache:master Mar 19, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants