Skip to content

[fix](routine-load) fix routine load lag is negative (#33846)#37449

Merged
dataroaring merged 1 commit intoapache:branch-2.0from
sollhui:pick_33846_to_be6ba909f8ddb01620c1f2ce0b482839f3af42b7
Jul 8, 2024
Merged

[fix](routine-load) fix routine load lag is negative (#33846)#37449
dataroaring merged 1 commit intoapache:branch-2.0from
sollhui:pick_33846_to_be6ba909f8ddb01620c1f2ce0b482839f3af42b7

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Jul 8, 2024

pick #33846

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@github-actions github-actions bot added the area/load Issues or PRs related to all kinds of load label Jul 8, 2024
@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

2 similar comments
@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

1 similar comment
@sollhui
Copy link
Contributor Author

sollhui commented Jul 8, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50045 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit d2f5ed30510b531986267631ff2791ad1f255b12, data reload: false

------ Round 1 ----------------------------------
q1	17628	4473	4403	4403
q2	2108	163	157	157
q3	10249	1906	1938	1906
q4	10126	1257	1322	1257
q5	8449	3959	3923	3923
q6	234	157	130	130
q7	2083	1581	1640	1581
q8	9307	2768	2742	2742
q9	10944	10211	10245	10211
q10	8632	3498	3535	3498
q11	444	260	244	244
q12	472	310	310	310
q13	18338	4002	4043	4002
q14	353	325	338	325
q15	502	471	479	471
q16	698	570	577	570
q17	1137	994	959	959
q18	7324	6945	6806	6806
q19	1794	1685	1638	1638
q20	542	320	304	304
q21	4478	4172	4206	4172
q22	528	448	436	436
Total cold run time: 116370 ms
Total hot run time: 50045 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4343	4303	4376	4303
q2	318	234	224	224
q3	4198	4194	4150	4150
q4	2760	2761	2780	2761
q5	7217	7146	7189	7146
q6	241	122	126	122
q7	3226	2850	2859	2850
q8	4417	4453	4512	4453
q9	16833	16883	16717	16717
q10	4263	4299	4246	4246
q11	737	709	672	672
q12	1027	872	873	872
q13	7000	3739	3764	3739
q14	456	433	438	433
q15	508	455	456	455
q16	738	677	693	677
q17	3788	3848	3866	3848
q18	8809	8773	8929	8773
q19	1731	1749	1667	1667
q20	2351	2190	2142	2142
q21	8524	8505	8452	8452
q22	1043	999	999	999
Total cold run time: 84528 ms
Total hot run time: 79701 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 203486 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit d2f5ed30510b531986267631ff2791ad1f255b12, data reload: false

query1	922	419	381	381
query2	6546	2775	2729	2729
query3	6919	206	204	204
query4	20999	18010	17958	17958
query5	19729	6545	6536	6536
query6	284	228	229	228
query7	4162	297	318	297
query8	441	401	399	399
query9	3082	2685	2627	2627
query10	414	311	304	304
query11	11333	10684	10742	10684
query12	126	77	77	77
query13	5614	705	678	678
query14	17994	13431	13615	13431
query15	404	239	248	239
query16	6463	289	262	262
query17	1703	1448	911	911
query18	2287	416	426	416
query19	205	149	154	149
query20	75	76	81	76
query21	193	100	97	97
query22	5304	5086	5025	5025
query23	32392	31936	32039	31936
query24	6999	6548	6535	6535
query25	545	437	421	421
query26	528	167	161	161
query27	1874	294	300	294
query28	6244	2344	2301	2301
query29	2942	2799	2786	2786
query30	250	164	169	164
query31	911	791	779	779
query32	70	66	63	63
query33	392	249	258	249
query34	847	482	488	482
query35	1121	926	896	896
query36	1246	1110	1344	1110
query37	92	59	64	59
query38	3047	2883	2956	2883
query39	1373	1328	1318	1318
query40	210	97	98	97
query41	47	43	39	39
query42	86	87	88	87
query43	855	648	684	648
query44	1126	723	706	706
query45	246	234	233	233
query46	1249	989	975	975
query47	1786	2122	1879	1879
query48	1017	708	706	706
query49	630	370	362	362
query50	874	666	614	614
query51	4750	4651	4683	4651
query52	94	90	82	82
query53	455	332	319	319
query54	2660	2478	2461	2461
query55	104	78	86	78
query56	216	224	216	216
query57	1147	1127	1127	1127
query58	225	206	190	190
query59	4085	3998	3839	3839
query60	227	196	218	196
query61	102	93	94	93
query62	837	460	489	460
query63	494	351	357	351
query64	2582	1524	1535	1524
query65	3633	3540	3622	3540
query66	807	381	386	381
query67	15331	15474	14953	14953
query68	8712	657	658	657
query69	566	343	345	343
query70	1702	1390	1324	1324
query71	406	311	312	311
query72	6538	3489	3506	3489
query73	736	316	322	316
query74	6263	5879	5791	5791
query75	5127	3708	3645	3645
query76	5288	1152	1156	1152
query77	887	261	265	261
query78	12442	11577	12121	11577
query79	8661	621	634	621
query80	1451	406	400	400
query81	484	236	231	231
query82	1558	102	100	100
query83	176	134	129	129
query84	262	72	69	69
query85	891	335	328	328
query86	321	340	288	288
query87	3263	2993	3011	2993
query88	5106	2313	2306	2306
query89	386	281	301	281
query90	1916	208	203	203
query91	171	141	148	141
query92	58	60	54	54
query93	5682	574	592	574
query94	701	210	211	210
query95	1115	1085	1058	1058
query96	639	322	314	314
query97	6492	6356	6357	6356
query98	189	173	173	173
query99	2916	913	839	839
Total cold run time: 313052 ms
Total hot run time: 203486 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.57 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit d2f5ed30510b531986267631ff2791ad1f255b12, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.04
query4	1.80	0.06	0.06
query5	0.56	0.53	0.52
query6	1.23	0.62	0.62
query7	0.02	0.01	0.01
query8	0.03	0.02	0.02
query9	0.51	0.48	0.49
query10	0.53	0.54	0.53
query11	0.11	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.60	0.60
query14	0.78	0.78	0.79
query15	0.77	0.76	0.75
query16	0.36	0.37	0.39
query17	1.02	1.02	1.01
query18	0.23	0.26	0.22
query19	1.92	1.87	1.88
query20	0.01	0.02	0.01
query21	15.46	0.54	0.55
query22	2.00	1.65	1.60
query23	17.19	1.02	1.10
query24	4.89	4.97	1.71
query25	0.40	0.09	0.06
query26	0.97	0.15	0.15
query27	0.04	0.04	0.03
query28	3.67	0.74	0.75
query29	12.64	2.35	2.34
query30	0.54	0.51	0.52
query31	2.81	0.40	0.39
query32	3.33	0.49	0.50
query33	3.05	3.06	3.06
query34	15.28	4.80	4.77
query35	4.85	4.85	4.83
query36	1.04	1.00	1.01
query37	0.06	0.05	0.04
query38	0.04	0.02	0.03
query39	0.02	0.01	0.02
query40	0.17	0.14	0.13
query41	0.06	0.01	0.02
query42	0.02	0.01	0.01
query43	0.03	0.01	0.02
Total cold run time: 99.52 s
Total hot run time: 31.57 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit d2f5ed30510b531986267631ff2791ad1f255b12 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.6 seconds inserted 10000000 Rows, about 462K ops/s

@dataroaring dataroaring merged commit d88af6c into apache:branch-2.0 Jul 8, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area/load Issues or PRs related to all kinds of load

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants