Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](routine-load) fix routine load lag is negative #33846

Merged
merged 1 commit into from
Apr 24, 2024
Merged

[fix](routine-load) fix routine load lag is negative #33846

merged 1 commit into from
Apr 24, 2024

Conversation

sollhui
Copy link
Contributor

@sollhui sollhui commented Apr 18, 2024 •
edited
Loading

Proposed changes

If the amount of data consumed during the consumption process exceeds the maximum offset of obtaining kafka before task scheduling, it will become a negative number.

To solve this problem, obtaining the maximum offset during transaction submission has two benefits:

  1. Avoid lag being negative
  2. Reduce possible RPC to Kafka in function hasMoreDataToConsume, as offset has been updated in a timely manner.

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actions bot reviewed Apr 18, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/routine_load/routine_load_task_executor.cpp Outdated Show resolved Hide resolved
be/src/runtime/routine_load/routine_load_task_executor.cpp Outdated Show resolved Hide resolved
@sollhui
Copy link
Contributor Author

sollhui commented Apr 18, 2024

run buildall

github-actions[bot]
github-actions bot reviewed Apr 18, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/routine_load/routine_load_task_executor.cpp Outdated Show resolved Hide resolved
@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8909/25158)
Line Coverage: 27.12% (73221/269992)
Region Coverage: 26.24% (37814/144110)
Branch Coverage: 23.05% (19273/83610)
Coverage Report: http://coverage.selectdb-in.cc/coverage/beb937f56e994df2a524d99a03957aa3d350909b_beb937f56e994df2a524d99a03957aa3d350909b/report/index.html

@sollhui
Copy link
Contributor Author

sollhui commented Apr 18, 2024

run buildall

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.41% (8909/25158)
Line Coverage: 27.12% (73230/269995)
Region Coverage: 26.25% (37825/144110)
Branch Coverage: 23.05% (19275/83610)
Coverage Report: http://coverage.selectdb-in.cc/coverage/a9a9040bbad9d80596f92cb3d1af37757f2a5ef9_a9a9040bbad9d80596f92cb3d1af37757f2a5ef9/report/index.html

@sollhui
Copy link
Contributor Author

sollhui commented Apr 19, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@sollhui sollhui mentioned this pull request Apr 19, 2024
Merged
github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/routine_load/data_consumer.cpp Outdated Show resolved Hide resolved
be/src/runtime/routine_load/data_consumer.cpp Outdated Show resolved Hide resolved
be/src/runtime/routine_load/data_consumer_group.cpp Outdated Show resolved Hide resolved
github-actions[bot]
github-actions bot reviewed Apr 19, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/runtime/routine_load/data_consumer.cpp Outdated Show resolved Hide resolved
@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38649 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 613b034f8b025287eec0e13912689f17dd138208, data reload: false

------ Round 1 ----------------------------------
q1	17712	4432	4286	4286
q2	2021	194	184	184
q3	10522	1174	1214	1174
q4	10214	801	717	717
q5	7539	2688	2679	2679
q6	219	134	138	134
q7	1051	613	588	588
q8	9224	2092	2035	2035
q9	7540	6636	6536	6536
q10	8620	3543	3536	3536
q11	447	230	225	225
q12	471	218	217	217
q13	17755	2902	2919	2902
q14	275	224	237	224
q15	516	497	479	479
q16	524	406	379	379
q17	971	696	740	696
q18	7402	6787	6795	6787
q19	5536	1538	1530	1530
q20	658	322	297	297
q21	3489	2855	2745	2745
q22	363	299	303	299
Total cold run time: 113069 ms
Total hot run time: 38649 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4338	4240	4209	4209
q2	368	264	270	264
q3	2973	2750	2792	2750
q4	1894	1553	1588	1553
q5	5341	5344	5306	5306
q6	207	124	123	123
q7	2259	1893	1867	1867
q8	3211	3379	3325	3325
q9	8506	8478	8605	8478
q10	4051	3879	3989	3879
q11	622	495	500	495
q12	803	625	657	625
q13	16393	3212	3138	3138
q14	329	276	300	276
q15	515	508	482	482
q16	499	450	451	450
q17	1824	1552	1495	1495
q18	8029	8189	7814	7814
q19	1680	1603	1523	1523
q20	2059	1865	1841	1841
q21	8593	5129	5082	5082
q22	550	465	474	465
Total cold run time: 75044 ms
Total hot run time: 55440 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185780 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 613b034f8b025287eec0e13912689f17dd138208, data reload: false

query1	895	381	375	375
query2	6225	2694	2310	2310
query3	6657	217	204	204
query4	25211	21239	21166	21166
query5	4176	399	403	399
query6	265	181	172	172
query7	4584	308	283	283
query8	247	191	189	189
query9	8535	2321	2289	2289
query10	438	240	252	240
query11	14657	14221	14181	14181
query12	140	94	85	85
query13	1640	384	354	354
query14	9217	7960	7840	7840
query15	263	178	179	178
query16	8167	263	263	263
query17	1935	552	546	546
query18	2109	272	260	260
query19	319	144	147	144
query20	94	83	84	83
query21	200	123	123	123
query22	5034	4922	4865	4865
query23	34074	33327	33178	33178
query24	10979	3008	3014	3008
query25	573	386	382	382
query26	704	160	159	159
query27	2329	360	373	360
query28	5919	2079	2127	2079
query29	854	618	617	617
query30	286	182	177	177
query31	973	779	759	759
query32	94	53	50	50
query33	649	249	255	249
query34	899	491	506	491
query35	848	721	721	721
query36	1083	951	944	944
query37	108	72	66	66
query38	3472	3408	3366	3366
query39	1062	1033	987	987
query40	181	123	131	123
query41	45	42	43	42
query42	108	93	98	93
query43	585	540	551	540
query44	1094	731	778	731
query45	283	270	254	254
query46	1090	742	735	735
query47	2068	1979	1974	1974
query48	370	299	324	299
query49	883	384	395	384
query50	783	394	393	393
query51	6967	6765	6718	6718
query52	107	88	89	88
query53	339	270	272	270
query54	299	263	236	236
query55	77	71	74	71
query56	242	229	227	227
query57	1202	1154	1128	1128
query58	218	205	206	205
query59	3222	3232	3138	3138
query60	261	228	233	228
query61	105	105	102	102
query62	582	443	436	436
query63	303	275	276	275
query64	4692	3910	4053	3910
query65	3054	3009	3008	3008
query66	746	335	330	330
query67	15591	15074	14816	14816
query68	6781	547	550	547
query69	553	310	313	310
query70	1222	1198	1163	1163
query71	1477	1267	1265	1265
query72	6596	2739	2572	2572
query73	723	318	319	318
query74	6941	6486	6361	6361
query75	3853	2585	2637	2585
query76	4775	1007	979	979
query77	626	261	259	259
query78	10893	10243	10246	10243
query79	11940	515	510	510
query80	2041	429	427	427
query81	521	240	239	239
query82	821	95	93	93
query83	195	159	159	159
query84	267	82	81	81
query85	1360	258	311	258
query86	461	284	291	284
query87	3472	3264	3256	3256
query88	5318	2305	2318	2305
query89	550	376	369	369
query90	1986	182	175	175
query91	122	96	98	96
query92	60	46	45	45
query93	7882	514	504	504
query94	1242	180	177	177
query95	1097	1097	1085	1085
query96	606	268	264	264
query97	3148	2926	2939	2926
query98	242	220	213	213
query99	1227	843	816	816
Total cold run time: 300409 ms
Total hot run time: 185780 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.81 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 613b034f8b025287eec0e13912689f17dd138208, data reload: false

query1	0.04	0.03	0.04
query2	0.08	0.03	0.04
query3	0.23	0.05	0.05
query4	1.67	0.07	0.07
query5	0.48	0.52	0.49
query6	1.47	0.72	0.73
query7	0.02	0.02	0.01
query8	0.05	0.04	0.05
query9	0.55	0.50	0.49
query10	0.55	0.55	0.55
query11	0.15	0.11	0.12
query12	0.14	0.11	0.12
query13	0.61	0.58	0.58
query14	0.77	0.76	0.77
query15	0.83	0.81	0.81
query16	0.36	0.36	0.34
query17	0.95	1.00	1.02
query18	0.21	0.24	0.22
query19	1.77	1.65	1.65
query20	0.01	0.02	0.01
query21	15.40	0.65	0.64
query22	4.70	8.12	1.45
query23	18.27	1.34	1.30
query24	1.94	0.21	0.22
query25	0.14	0.07	0.09
query26	0.26	0.17	0.17
query27	0.08	0.07	0.08
query28	13.34	1.01	0.98
query29	12.63	3.28	3.27
query30	0.26	0.08	0.06
query31	2.85	0.38	0.37
query32	3.29	0.46	0.46
query33	2.76	2.84	2.86
query34	17.02	4.36	4.43
query35	4.45	4.45	4.48
query36	0.65	0.48	0.46
query37	0.19	0.16	0.15
query38	0.15	0.14	0.15
query39	0.05	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.05
query42	0.06	0.04	0.05
query43	0.04	0.03	0.04
Total cold run time: 109.73 s
Total hot run time: 29.81 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 613b034f8b025287eec0e13912689f17dd138208 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      33 seconds loaded 861443392 Bytes, about 24 MB/s
Insert into select:       13.1 seconds inserted 10000000 Rows, about 763K ops/s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.39% (8915/25189)
Line Coverage: 27.11% (73257/270223)
Region Coverage: 26.24% (37837/144196)
Branch Coverage: 23.06% (19276/83608)
Coverage Report: http://coverage.selectdb-in.cc/coverage/613b034f8b025287eec0e13912689f17dd138208_613b034f8b025287eec0e13912689f17dd138208/report/index.html

@dataroaring
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 38712 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ebb6755e9b5fff35546e3c432ffede822631c7b5, data reload: false

------ Round 1 ----------------------------------
q1	17678	4323	4306	4306
q2	2020	190	189	189
q3	10518	1181	1216	1181
q4	10194	773	832	773
q5	7525	2762	2687	2687
q6	214	131	134	131
q7	1018	609	579	579
q8	9230	2089	2069	2069
q9	7450	6621	6552	6552
q10	8551	3520	3565	3520
q11	451	235	235	235
q12	445	229	226	226
q13	18673	2955	2956	2955
q14	266	234	237	234
q15	545	478	475	475
q16	515	390	378	378
q17	978	722	718	718
q18	7559	6876	6687	6687
q19	7248	1572	1531	1531
q20	638	305	303	303
q21	3468	2793	2678	2678
q22	368	313	305	305
Total cold run time: 115552 ms
Total hot run time: 38712 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4366	4241	4221	4221
q2	366	271	270	270
q3	3030	2784	2721	2721
q4	1884	1587	1612	1587
q5	5368	5376	5316	5316
q6	209	124	125	124
q7	2256	1879	1915	1879
q8	3214	3366	3373	3366
q9	8652	8645	8712	8645
q10	3865	3710	3686	3686
q11	572	495	494	494
q12	753	585	581	581
q13	16348	2948	2915	2915
q14	305	260	263	260
q15	512	483	465	465
q16	483	408	427	408
q17	1770	1496	1458	1458
q18	7691	7472	7405	7405
q19	1660	1513	1541	1513
q20	1943	1752	1734	1734
q21	4852	4857	4800	4800
q22	540	478	466	466
Total cold run time: 70639 ms
Total hot run time: 54314 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184165 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ebb6755e9b5fff35546e3c432ffede822631c7b5, data reload: false

query1	903	377	361	361
query2	7176	2681	2346	2346
query3	6656	206	210	206
query4	23711	21203	21455	21203
query5	4115	414	422	414
query6	269	189	170	170
query7	4589	284	289	284
query8	245	202	195	195
query9	8468	2358	2315	2315
query10	584	243	248	243
query11	14754	14281	14360	14281
query12	135	92	87	87
query13	1631	356	361	356
query14	9941	7501	6953	6953
query15	266	177	183	177
query16	8127	256	254	254
query17	1933	567	569	567
query18	2105	276	265	265
query19	194	146	151	146
query20	90	84	88	84
query21	206	124	125	124
query22	5007	4893	4869	4869
query23	33844	33263	33270	33263
query24	10797	3013	2938	2938
query25	615	370	371	370
query26	1155	153	156	153
query27	2834	311	318	311
query28	7287	2038	2011	2011
query29	846	603	620	603
query30	286	177	178	177
query31	990	754	733	733
query32	95	54	55	54
query33	746	257	268	257
query34	1051	485	492	485
query35	849	701	677	677
query36	1050	922	903	903
query37	141	69	71	69
query38	3373	3244	3167	3167
query39	1590	1518	1525	1518
query40	208	129	133	129
query41	47	43	44	43
query42	104	96	94	94
query43	587	560	555	555
query44	1256	736	730	730
query45	271	267	262	262
query46	1078	715	731	715
query47	1918	1823	1830	1823
query48	364	302	291	291
query49	1071	394	398	394
query50	758	391	373	373
query51	6897	6627	6554	6554
query52	102	86	96	86
query53	340	280	301	280
query54	300	232	238	232
query55	76	73	73	73
query56	248	221	213	213
query57	1201	1129	1120	1120
query58	222	194	199	194
query59	3572	3398	3692	3398
query60	254	232	225	225
query61	90	88	90	88
query62	656	467	443	443
query63	302	275	275	275
query64	5327	3734	3899	3734
query65	3067	3100	3014	3014
query66	951	330	338	330
query67	15523	15229	14881	14881
query68	5229	538	533	533
query69	485	298	302	298
query70	1241	1151	1152	1151
query71	1389	1265	1277	1265
query72	6387	2590	2440	2440
query73	717	323	319	319
query74	6880	6461	6404	6404
query75	3370	2595	2641	2595
query76	2902	967	965	965
query77	483	263	268	263
query78	10884	10204	10135	10135
query79	2668	526	507	507
query80	2077	428	431	428
query81	532	242	239	239
query82	698	99	96	96
query83	298	179	170	170
query84	272	85	83	83
query85	1974	302	318	302
query86	489	303	293	293
query87	3533	3291	3304	3291
query88	4354	2330	2326	2326
query89	474	369	368	368
query90	1983	178	180	178
query91	134	108	108	108
query92	65	49	47	47
query93	2508	508	500	500
query94	1265	183	181	181
query95	394	365	286	286
query96	587	261	259	259
query97	3139	2920	2928	2920
query98	239	214	213	213
query99	1206	864	868	864
Total cold run time: 284300 ms
Total hot run time: 184165 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ebb6755e9b5fff35546e3c432ffede822631c7b5, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.50	0.49	0.49
query6	1.48	0.73	0.72
query7	0.02	0.01	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.49
query10	0.54	0.56	0.56
query11	0.16	0.11	0.12
query12	0.14	0.12	0.12
query13	0.60	0.59	0.58
query14	0.76	0.77	0.76
query15	0.83	0.82	0.80
query16	0.36	0.36	0.37
query17	1.04	0.95	0.99
query18	0.24	0.24	0.25
query19	1.76	1.69	1.74
query20	0.02	0.01	0.02
query21	15.43	0.65	0.65
query22	4.28	7.01	2.12
query23	18.27	1.39	1.22
query24	1.90	0.26	0.19
query25	0.14	0.09	0.09
query26	0.25	0.16	0.15
query27	0.07	0.07	0.08
query28	13.28	0.99	0.99
query29	12.60	3.30	3.24
query30	0.26	0.06	0.06
query31	2.85	0.37	0.37
query32	3.30	0.47	0.46
query33	2.80	2.80	2.84
query34	17.12	4.40	4.35
query35	4.52	4.42	4.47
query36	0.63	0.46	0.47
query37	0.19	0.16	0.16
query38	0.14	0.14	0.15
query39	0.04	0.04	0.04
query40	0.18	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.04	0.05
query43	0.04	0.04	0.03
Total cold run time: 109.54 s
Total hot run time: 30.35 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 35.36% (8911/25204)
Line Coverage: 27.09% (73274/270532)
Region Coverage: 26.21% (37840/144354)
Branch Coverage: 23.03% (19273/83684)
Coverage Report: http://coverage.selectdb-in.cc/coverage/ebb6755e9b5fff35546e3c432ffede822631c7b5_ebb6755e9b5fff35546e3c432ffede822631c7b5/report/index.html

@sollhui
Copy link
Contributor Author

sollhui commented Apr 22, 2024

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39008 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 67695618c16745ad4210f3e61da0b55efdb40d1f, data reload: false

------ Round 1 ----------------------------------
q1	17634	7220	4385	4385
q2	2468	202	187	187
q3	10909	1124	1148	1124
q4	12020	787	866	787
q5	7859	2736	2636	2636
q6	221	134	136	134
q7	1012	629	613	613
q8	10113	2077	2079	2077
q9	7453	6722	6694	6694
q10	8851	3642	3608	3608
q11	478	260	229	229
q12	452	244	218	218
q13	18028	3086	3018	3018
q14	264	223	228	223
q15	514	491	478	478
q16	526	404	378	378
q17	961	686	741	686
q18	7407	6902	6617	6617
q19	6993	1535	1515	1515
q20	667	335	300	300
q21	3531	2793	2795	2793
q22	369	308	312	308
Total cold run time: 118730 ms
Total hot run time: 39008 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4396	4254	4283	4254
q2	377	278	261	261
q3	3018	2759	2741	2741
q4	1832	1563	1568	1563
q5	5350	5318	5330	5318
q6	209	122	130	122
q7	2203	1857	1906	1857
q8	3200	3350	3333	3333
q9	8565	8588	8551	8551
q10	3852	3727	3702	3702
q11	582	500	483	483
q12	731	612	586	586
q13	17340	2939	2955	2939
q14	287	274	272	272
q15	526	479	475	475
q16	469	414	438	414
q17	1780	1480	1464	1464
q18	7634	7433	7327	7327
q19	1629	1544	1546	1544
q20	1946	1773	1755	1755
q21	4944	4711	4762	4711
q22	530	452	470	452
Total cold run time: 71400 ms
Total hot run time: 54124 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 184905 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 67695618c16745ad4210f3e61da0b55efdb40d1f, data reload: false

query1	894	391	374	374
query2	7533	2417	2389	2389
query3	6663	206	207	206
query4	24706	21250	21300	21250
query5	4159	420	427	420
query6	273	185	168	168
query7	4589	289	293	289
query8	241	184	187	184
query9	8401	2333	2312	2312
query10	607	247	253	247
query11	14727	14193	14355	14193
query12	142	100	90	90
query13	1640	369	362	362
query14	10120	7967	8112	7967
query15	280	197	189	189
query16	8178	272	268	268
query17	1955	595	576	576
query18	1977	288	284	284
query19	343	190	160	160
query20	94	84	87	84
query21	199	132	126	126
query22	5096	4806	4888	4806
query23	33828	33398	33212	33212
query24	11583	2959	3016	2959
query25	653	376	388	376
query26	1733	154	153	153
query27	2825	315	325	315
query28	7445	1974	1971	1971
query29	1004	606	589	589
query30	270	179	174	174
query31	943	719	750	719
query32	94	55	59	55
query33	768	256	246	246
query34	1014	479	498	479
query35	828	704	687	687
query36	1048	926	904	904
query37	274	72	72	72
query38	3431	3249	3221	3221
query39	1589	1542	1570	1542
query40	268	125	128	125
query41	46	45	48	45
query42	103	99	95	95
query43	562	509	540	509
query44	1232	727	728	727
query45	293	269	269	269
query46	1074	736	717	717
query47	1937	1840	1851	1840
query48	372	291	290	290
query49	1162	397	385	385
query50	768	387	390	387
query51	6654	6531	6607	6531
query52	98	96	93	93
query53	351	288	281	281
query54	324	240	241	240
query55	81	79	74	74
query56	244	229	219	219
query57	1218	1142	1119	1119
query58	234	195	211	195
query59	3466	3194	3105	3105
query60	246	232	232	232
query61	89	88	90	88
query62	660	442	455	442
query63	303	282	285	282
query64	6204	3739	3704	3704
query65	3222	3033	3030	3030
query66	1338	353	358	353
query67	15414	14890	14842	14842
query68	5323	543	537	537
query69	483	299	310	299
query70	1220	1151	1195	1151
query71	1379	1267	1306	1267
query72	6353	2637	2418	2418
query73	706	319	319	319
query74	6839	6417	6444	6417
query75	3399	2662	2626	2626
query76	3330	1006	982	982
query77	412	261	271	261
query78	11041	10225	10220	10220
query79	2984	499	515	499
query80	1982	439	424	424
query81	536	242	240	240
query82	773	94	100	94
query83	278	171	168	168
query84	264	91	82	82
query85	2090	287	366	287
query86	483	289	289	289
query87	3427	3270	3291	3270
query88	4577	2318	2301	2301
query89	476	373	372	372
query90	2011	181	182	181
query91	127	97	98	97
query92	63	47	46	46
query93	4193	520	500	500
query94	1253	181	183	181
query95	390	302	304	302
query96	602	268	261	261
query97	3128	2921	2943	2921
query98	250	220	216	216
query99	1238	852	840	840
Total cold run time: 291720 ms
Total hot run time: 184905 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 183175 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c89355b0208893c69006b2bfb5a3d0e6e1593575, data reload: false

query1	902	376	373	373
query2	6494	2320	2390	2320
query3	6661	211	213	211
query4	22703	21281	21436	21281
query5	4122	429	412	412
query6	267	173	170	170
query7	4588	293	285	285
query8	240	189	187	187
query9	8505	2266	2273	2266
query10	607	255	256	255
query11	14713	14074	14147	14074
query12	139	96	92	92
query13	1649	379	370	370
query14	9494	7997	7026	7026
query15	291	183	185	183
query16	8190	265	267	265
query17	1933	601	560	560
query18	2117	284	283	283
query19	284	151	151	151
query20	91	86	84	84
query21	203	121	124	121
query22	4956	4783	4825	4783
query23	33905	33075	33161	33075
query24	12083	2898	2984	2898
query25	645	372	374	372
query26	1776	152	161	152
query27	3026	312	312	312
query28	7633	1951	1947	1947
query29	1023	589	592	589
query30	298	179	174	174
query31	975	724	738	724
query32	97	55	55	55
query33	768	253	248	248
query34	1038	468	475	468
query35	841	682	699	682
query36	1062	936	885	885
query37	252	72	69	69
query38	3314	3143	3179	3143
query39	1583	1531	1541	1531
query40	274	123	126	123
query41	47	43	42	42
query42	106	97	99	97
query43	555	550	555	550
query44	1259	732	719	719
query45	278	259	260	259
query46	1089	699	695	695
query47	1888	1832	1811	1811
query48	370	296	289	289
query49	1205	391	403	391
query50	761	379	390	379
query51	6689	6650	6624	6624
query52	104	90	97	90
query53	352	281	283	281
query54	317	243	259	243
query55	75	73	71	71
query56	242	229	222	222
query57	1191	1129	1113	1113
query58	223	195	194	194
query59	3351	3027	3133	3027
query60	249	232	237	232
query61	90	90	88	88
query62	652	453	445	445
query63	303	276	279	276
query64	6292	3626	3793	3626
query65	3080	3069	3076	3069
query66	1388	335	335	335
query67	15255	14833	15087	14833
query68	5240	529	534	529
query69	473	295	301	295
query70	1187	1169	1180	1169
query71	1396	1272	1271	1271
query72	6552	2603	2434	2434
query73	706	319	319	319
query74	6894	6371	6463	6371
query75	3351	2665	2583	2583
query76	2967	970	991	970
query77	404	267	269	267
query78	11023	10087	10198	10087
query79	2716	524	520	520
query80	2031	425	439	425
query81	536	240	240	240
query82	777	94	100	94
query83	273	167	171	167
query84	272	87	81	81
query85	2067	272	270	270
query86	488	311	301	301
query87	3485	3295	3257	3257
query88	4490	2408	2407	2407
query89	476	374	366	366
query90	2023	184	185	184
query91	130	100	102	100
query92	55	48	48	48
query93	4858	503	493	493
query94	1236	182	181	181
query95	384	312	312	312
query96	610	271	269	269
query97	3134	2898	2935	2898
query98	245	233	218	218
query99	1207	849	888	849
Total cold run time: 288839 ms
Total hot run time: 183175 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.41 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c89355b0208893c69006b2bfb5a3d0e6e1593575, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.04
query3	0.23	0.05	0.04
query4	1.68	0.07	0.07
query5	0.50	0.49	0.51
query6	1.43	0.72	0.74
query7	0.03	0.02	0.01
query8	0.05	0.04	0.04
query9	0.55	0.49	0.49
query10	0.55	0.56	0.55
query11	0.17	0.11	0.11
query12	0.14	0.12	0.12
query13	0.61	0.58	0.59
query14	0.76	0.78	0.78
query15	0.82	0.80	0.81
query16	0.35	0.36	0.38
query17	1.01	1.02	0.93
query18	0.21	0.25	0.22
query19	1.76	1.69	1.67
query20	0.01	0.01	0.01
query21	15.41	0.66	0.65
query22	3.89	7.13	2.14
query23	18.24	1.30	1.18
query24	1.87	0.24	0.22
query25	0.15	0.10	0.08
query26	0.26	0.17	0.16
query27	0.08	0.08	0.09
query28	13.34	0.99	0.97
query29	12.65	3.26	3.24
query30	0.26	0.06	0.05
query31	2.83	0.37	0.38
query32	3.28	0.46	0.46
query33	2.77	2.90	2.81
query34	17.22	4.42	4.45
query35	4.57	4.53	4.47
query36	0.64	0.46	0.48
query37	0.19	0.16	0.15
query38	0.16	0.14	0.14
query39	0.04	0.03	0.04
query40	0.17	0.14	0.14
query41	0.09	0.05	0.06
query42	0.06	0.05	0.05
query43	0.04	0.03	0.03
Total cold run time: 109.19 s
Total hot run time: 30.41 s

liaoxin01
liaoxin01 approved these changes Apr 22, 2024
View reviewed changes
Copy link
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@liaoxin01 liaoxin01 self-requested a review April 23, 2024 02:18
XuJianxu
XuJianxu approved these changes Apr 23, 2024
View reviewed changes
Copy link
Contributor

@XuJianxu XuJianxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit df07e1a into apache:master Apr 24, 2024
28 of 34 checks passed
yiguolei pushed a commit that referenced this pull request Apr 24, 2024
@sollhui @Doris-Extras
[fix](routine-load) fix routine load lag is negative (#33846)
f88093e
dataroaring pushed a commit that referenced this pull request Apr 24, 2024
@sollhui @dataroaring
[fix](routine-load) fix routine load lag is negative (#33846)
16998e0
sollhui added a commit to sollhui/doris that referenced this pull request Apr 25, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
9907a74
@sollhui sollhui mentioned this pull request Apr 25, 2024
Merged
yiguolei pushed a commit that referenced this pull request Apr 25, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (#34113)
b873be6 
* [fix](routine-load) fix routine load lag is negative (#33846)

* fix merge error
sollhui added a commit to sollhui/doris that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
f3b90a2
@sollhui sollhui mentioned this pull request Jul 8, 2024
Merged
sollhui added a commit to sollhui/doris that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
d3eada6
sollhui added a commit to sollhui/doris that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
a3d93d9
sollhui added a commit to sollhui/doris that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
1bb543c
sollhui added a commit to sollhui/doris that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846)
d2f5ed3
dataroaring pushed a commit that referenced this pull request Jul 8, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (#33846) (#37449)
d88af6c 
pick #33846
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@sollhui
[fix](routine-load) fix routine load lag is negative (apache#33846) (a…
0963b68 
…pache#37449)

pick apache#33846
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@XuJianxu XuJianxu XuJianxu approved these changes

@liaoxin01 liaoxin01 liaoxin01 approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.13-merged meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@sollhui @doris-robot @dataroaring @liaoxin01 @XuJianxu @yiguolei