Skip to content

[fix](be) explicitly close stream load recorder RocksDB on shutdown#63004

Open
sollhui wants to merge 1 commit intoapache:masterfrom
sollhui:fix_stream_load_recorder_shutdown
Open

[fix](be) explicitly close stream load recorder RocksDB on shutdown#63004
sollhui wants to merge 1 commit intoapache:masterfrom
sollhui:fix_stream_load_recorder_shutdown

Conversation

@sollhui
Copy link
Copy Markdown
Contributor

@sollhui sollhui commented May 6, 2026

What problem does this PR solve?

Problem Summary:

Explicitly close the RocksDB instance used by StreamLoadRecorder during destruction.

Previously, StreamLoadRecorder::~StreamLoadRecorder() only destroyed column family handles and synced WAL, then relied on DBWithTTL destruction to close RocksDB implicitly. If BE failed during startup, for example because be_port was already occupied, shutdown cleanup could enter RocksDB DBImpl::CloseHelper() from the implicit destructor path and crash while unscheduling background work.

This patch makes the shutdown sequence explicit:

  • Sync WAL before closing
  • Cancel RocksDB background work and wait for completion
  • Destroy column family handles
  • Close the DB explicitly
  • Reset the DB pointer after close

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 6, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

liaoxin01
liaoxin01 previously approved these changes May 7, 2026
View reviewed changes
Copy link
Copy Markdown
Contributor

@liaoxin01 liaoxin01 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01
Copy link
Copy Markdown
Contributor

liaoxin01 commented May 7, 2026

run buildall

@github-actions github-actions Bot added the approved Indicates a PR has been approved by one committer. label May 7, 2026
@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Copy Markdown
Contributor

github-actions Bot commented May 7, 2026

PR approved by anyone and no changes requested.

@sollhui sollhui force-pushed the fix_stream_load_recorder_shutdown branch from e478eb4 to 2f9a376 Compare May 9, 2026 03:03
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 9, 2026

run buildall

@github-actions github-actions Bot removed the approved Indicates a PR has been approved by one committer. label May 9, 2026
@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-H: Total hot run time: 29363 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2f9a376ec5b025e8989ca5ed4b57139913d74dd4, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17792	3900	3818	3818
q2	q3	10711	877	604	604
q4	4663	458	348	348
q5	7440	1340	1151	1151
q6	188	169	136	136
q7	926	948	758	758
q8	9581	1419	1266	1266
q9	6092	5388	5372	5372
q10	6298	2087	1829	1829
q11	495	264	253	253
q12	687	420	298	298
q13	18174	3345	2715	2715
q14	288	282	260	260
q15	q16	904	868	796	796
q17	1002	1053	716	716
q18	6399	5766	5526	5526
q19	1397	1183	1072	1072
q20	506	403	257	257
q21	4738	2309	1885	1885
q22	421	361	303	303
Total cold run time: 98702 ms
Total hot run time: 29363 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4160	4066	4079	4066
q2	q3	4648	4769	4232	4232
q4	2120	2191	1399	1399
q5	4964	4916	5296	4916
q6	187	161	129	129
q7	2054	1995	1753	1753
q8	3432	3245	3210	3210
q9	8490	8555	8466	8466
q10	4506	4498	4193	4193
q11	625	434	386	386
q12	695	757	518	518
q13	3280	3664	2918	2918
q14	309	311	280	280
q15	q16	769	788	693	693
q17	1339	1315	1342	1315
q18	8083	7089	7031	7031
q19	1178	1167	1145	1145
q20	2246	2204	1951	1951
q21	6134	5434	4903	4903
q22	556	513	481	481
Total cold run time: 59775 ms
Total hot run time: 53985 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-DS: Total hot run time: 171346 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2f9a376ec5b025e8989ca5ed4b57139913d74dd4, data reload: false

query5	4345	654	522	522
query6	342	239	205	205
query7	4222	546	305	305
query8	323	236	224	224
query9	8827	4006	4023	4006
query10	463	337	294	294
query11	6034	2372	2194	2194
query12	188	131	137	131
query13	1314	614	428	428
query14	6804	5327	5027	5027
query14_1	4343	4359	4299	4299
query15	209	198	191	191
query16	995	456	410	410
query17	1350	758	627	627
query18	2731	481	350	350
query19	268	205	163	163
query20	143	136	138	136
query21	210	141	120	120
query22	13654	13820	13380	13380
query23	17136	16338	16624	16338
query23_1	16426	16222	16389	16222
query24	8089	1839	1392	1392
query24_1	1406	1392	1453	1392
query25	656	549	493	493
query26	1431	355	177	177
query27	3018	635	368	368
query28	4527	2052	2072	2052
query29	1140	716	605	605
query30	308	239	205	205
query31	1114	1056	934	934
query32	89	76	74	74
query33	577	366	305	305
query34	1145	1171	642	642
query35	816	808	698	698
query36	1318	1341	1152	1152
query37	151	107	90	90
query38	3191	3150	3014	3014
query39	943	907	884	884
query39_1	865	866	878	866
query40	247	159	137	137
query41	68	67	66	66
query42	120	109	109	109
query43	320	323	293	293
query44	
query45	209	204	195	195
query46	1046	1193	737	737
query47	2302	2286	2187	2187
query48	416	419	308	308
query49	616	550	415	415
query50	704	292	216	216
query51	4271	4287	4305	4287
query52	102	105	92	92
query53	252	274	201	201
query54	320	265	255	255
query55	88	88	81	81
query56	292	309	300	300
query57	1411	1410	1329	1329
query58	288	279	266	266
query59	1539	1633	1386	1386
query60	345	342	327	327
query61	160	157	150	150
query62	677	619	560	560
query63	257	205	214	205
query64	2364	807	713	713
query65	
query66	1676	500	391	391
query67	30240	29970	29826	29826
query68	
query69	455	340	304	304
query70	1028	1014	985	985
query71	317	288	265	265
query72	2990	2695	2485	2485
query73	868	758	425	425
query74	5078	4889	4735	4735
query75	2788	2664	2358	2358
query76	2300	1136	773	773
query77	418	433	366	366
query78	12915	12967	12190	12190
query79	1530	960	761	761
query80	1391	578	495	495
query81	492	286	243	243
query82	1256	166	127	127
query83	352	279	247	247
query84	270	143	118	118
query85	934	523	459	459
query86	449	309	311	309
query87	3413	3368	3231	3231
query88	3512	2671	2618	2618
query89	438	382	341	341
query90	1911	182	175	175
query91	181	174	144	144
query92	78	73	72	72
query93	965	956	562	562
query94	722	352	310	310
query95	663	454	345	345
query96	1059	825	342	342
query97	2720	2727	2563	2563
query98	247	227	226	226
query99	1128	1125	983	983
Total cold run time: 257193 ms
Total hot run time: 171346 ms

@sollhui sollhui force-pushed the fix_stream_load_recorder_shutdown branch from 2f9a376 to be1f8ad Compare May 9, 2026 04:46
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 9, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-H: Total hot run time: 29330 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be1f8ad878f533021ed449bc150067f63723154d, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17710	3840	3817	3817
q2	q3	10722	870	611	611
q4	4667	467	346	346
q5	7467	1323	1139	1139
q6	194	167	140	140
q7	928	950	769	769
q8	9496	1351	1294	1294
q9	6174	5443	5357	5357
q10	6360	2060	1806	1806
q11	476	263	254	254
q12	687	427	299	299
q13	18247	3308	2673	2673
q14	291	287	264	264
q15	q16	896	868	784	784
q17	1023	1024	744	744
q18	6467	5676	5589	5589
q19	1647	1318	1016	1016
q20	514	394	260	260
q21	4989	2294	1855	1855
q22	417	357	313	313
Total cold run time: 99372 ms
Total hot run time: 29330 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4206	4093	4107	4093
q2	q3	4634	4773	4156	4156
q4	2075	2184	1392	1392
q5	4948	4988	5282	4988
q6	196	163	132	132
q7	2054	1950	1670	1670
q8	3470	3220	3207	3207
q9	8554	8450	8353	8353
q10	4526	4494	4352	4352
q11	598	424	414	414
q12	701	753	531	531
q13	3229	3685	2877	2877
q14	309	320	285	285
q15	q16	747	771	727	727
q17	1346	1273	1405	1273
q18	8117	7136	7012	7012
q19	1161	1188	1193	1188
q20	2292	2288	1959	1959
q21	6195	5424	4924	4924
q22	616	514	424	424
Total cold run time: 59974 ms
Total hot run time: 53957 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-DS: Total hot run time: 171093 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be1f8ad878f533021ed449bc150067f63723154d, data reload: false

query5	4314	660	523	523
query6	354	232	205	205
query7	4219	559	305	305
query8	328	239	230	230
query9	8841	4060	4061	4060
query10	445	348	301	301
query11	5806	2371	2192	2192
query12	182	128	128	128
query13	1276	623	423	423
query14	6685	5351	5025	5025
query14_1	4360	4384	4363	4363
query15	216	207	196	196
query16	1025	469	381	381
query17	1182	794	639	639
query18	2775	489	365	365
query19	224	207	185	185
query20	141	136	131	131
query21	219	142	118	118
query22	13616	13948	14565	13948
query23	17450	16564	16337	16337
query23_1	16438	16275	16462	16275
query24	7823	1757	1335	1335
query24_1	1334	1335	1347	1335
query25	577	514	465	465
query26	1301	315	173	173
query27	2681	598	350	350
query28	4358	1948	1936	1936
query29	1017	656	532	532
query30	301	234	193	193
query31	1144	1072	949	949
query32	88	74	73	73
query33	522	360	301	301
query34	1154	1118	634	634
query35	783	821	668	668
query36	1316	1366	1130	1130
query37	147	100	87	87
query38	3208	3143	3058	3058
query39	927	918	899	899
query39_1	852	877	878	877
query40	231	152	134	134
query41	61	60	58	58
query42	108	107	111	107
query43	331	333	299	299
query44	
query45	211	196	206	196
query46	1091	1146	735	735
query47	2296	2250	2131	2131
query48	384	421	293	293
query49	640	520	432	432
query50	716	273	219	219
query51	4316	4413	4287	4287
query52	106	107	95	95
query53	260	284	203	203
query54	383	270	272	270
query55	91	91	84	84
query56	293	305	313	305
query57	1459	1433	1361	1361
query58	297	275	260	260
query59	1562	1669	1426	1426
query60	354	324	317	317
query61	191	154	156	154
query62	692	626	566	566
query63	252	207	213	207
query64	2375	848	724	724
query65	
query66	1695	521	407	407
query67	30226	30026	29811	29811
query68	
query69	464	325	307	307
query70	994	994	945	945
query71	304	279	268	268
query72	2991	2772	2426	2426
query73	824	758	405	405
query74	5097	4897	4744	4744
query75	2796	2653	2318	2318
query76	2294	1129	765	765
query77	429	417	351	351
query78	12949	12947	12189	12189
query79	1495	977	719	719
query80	1383	603	497	497
query81	528	284	234	234
query82	995	159	118	118
query83	348	290	249	249
query84	264	139	112	112
query85	913	511	445	445
query86	464	341	324	324
query87	3404	3365	3216	3216
query88	3592	2688	2673	2673
query89	433	381	335	335
query90	1960	183	180	180
query91	180	169	138	138
query92	73	79	74	74
query93	1109	964	587	587
query94	706	324	295	295
query95	676	452	347	347
query96	1088	787	340	340
query97	2702	2703	2567	2567
query98	236	232	227	227
query99	1114	1144	977	977
Total cold run time: 255915 ms
Total hot run time: 171093 ms

@sollhui sollhui force-pushed the fix_stream_load_recorder_shutdown branch from be1f8ad to eacdc0f Compare May 9, 2026 06:22
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 9, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

BE UT Coverage Report

Increment line coverage 100.00% (6/6) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.48% (20560/38445)
Line Coverage 37.11% (194120/523099)
Region Coverage 33.38% (150983/452364)
Branch Coverage 34.49% (66034/191477)

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-H: Total hot run time: 29487 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit eacdc0f07d5551a4d1d20fa5346cef8512057717, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17703	3907	3989	3907
q2	q3	10699	898	601	601
q4	4664	457	347	347
q5	7449	1320	1128	1128
q6	200	175	143	143
q7	944	957	739	739
q8	9689	1461	1312	1312
q9	6585	5382	5360	5360
q10	6309	2067	1795	1795
q11	481	269	263	263
q12	690	427	297	297
q13	18240	3325	2681	2681
q14	301	281	261	261
q15	q16	909	871	797	797
q17	997	1105	786	786
q18	6616	5781	5558	5558
q19	1199	1246	1069	1069
q20	519	388	259	259
q21	4549	2310	1881	1881
q22	427	360	303	303
Total cold run time: 99170 ms
Total hot run time: 29487 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4185	4152	4119	4119
q2	q3	4642	4819	4166	4166
q4	2103	2164	1404	1404
q5	4988	4959	5282	4959
q6	192	171	135	135
q7	2032	2181	1669	1669
q8	3533	3173	3224	3173
q9	8576	8479	8427	8427
q10	4498	4525	4291	4291
q11	649	440	422	422
q12	725	763	522	522
q13	3238	3584	2889	2889
q14	304	310	288	288
q15	q16	765	788	706	706
q17	1360	1365	1284	1284
q18	8205	7264	7080	7080
q19	1192	1174	1157	1157
q20	2243	2283	1960	1960
q21	6240	5549	5050	5050
q22	559	526	422	422
Total cold run time: 60229 ms
Total hot run time: 54123 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-DS: Total hot run time: 171971 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit eacdc0f07d5551a4d1d20fa5346cef8512057717, data reload: false

query5	4319	665	517	517
query6	324	231	212	212
query7	4231	599	314	314
query8	328	224	226	224
query9	8842	4106	4045	4045
query10	481	351	313	313
query11	5916	2394	2271	2271
query12	190	135	133	133
query13	1299	606	434	434
query14	6867	5409	5158	5158
query14_1	4391	4414	4397	4397
query15	212	207	189	189
query16	1017	470	458	458
query17	1427	791	649	649
query18	2760	506	375	375
query19	325	218	200	200
query20	138	135	130	130
query21	215	141	123	123
query22	13625	14090	14445	14090
query23	17496	16502	16215	16215
query23_1	16286	16346	16228	16228
query24	7702	1883	1429	1429
query24_1	1404	1369	1357	1357
query25	559	499	443	443
query26	1300	297	165	165
query27	2688	585	333	333
query28	4307	1973	1954	1954
query29	1009	632	547	547
query30	307	227	196	196
query31	1117	1058	945	945
query32	84	75	71	71
query33	545	346	308	308
query34	1157	1173	642	642
query35	778	791	666	666
query36	1334	1325	1135	1135
query37	155	105	90	90
query38	3194	3189	3073	3073
query39	937	907	885	885
query39_1	878	885	875	875
query40	232	160	140	140
query41	65	60	60	60
query42	111	109	106	106
query43	341	337	293	293
query44	
query45	201	198	192	192
query46	1067	1219	720	720
query47	2303	2307	2197	2197
query48	388	412	294	294
query49	643	533	432	432
query50	739	308	216	216
query51	4295	4424	4254	4254
query52	105	108	94	94
query53	245	279	211	211
query54	348	281	255	255
query55	93	91	90	90
query56	298	306	326	306
query57	1407	1403	1310	1310
query58	291	316	267	267
query59	1585	1652	1505	1505
query60	351	348	330	330
query61	155	152	170	152
query62	663	619	566	566
query63	245	208	219	208
query64	2381	830	687	687
query65	
query66	1687	507	403	403
query67	29447	30012	29934	29934
query68	
query69	457	346	301	301
query70	1053	967	1019	967
query71	320	279	280	279
query72	2880	2288	2524	2288
query73	844	761	438	438
query74	5058	4898	4780	4780
query75	2792	2689	2339	2339
query76	2320	1169	792	792
query77	445	443	370	370
query78	12957	13017	12501	12501
query79	1456	998	757	757
query80	720	610	525	525
query81	451	289	246	246
query82	1327	170	131	131
query83	363	297	266	266
query84	268	148	123	123
query85	929	605	515	515
query86	391	317	314	314
query87	3415	3373	3235	3235
query88	3584	2655	2667	2655
query89	438	383	341	341
query90	1851	196	192	192
query91	177	173	146	146
query92	83	79	69	69
query93	985	981	550	550
query94	556	336	304	304
query95	666	456	348	348
query96	1014	790	355	355
query97	2717	2729	2560	2560
query98	256	242	234	234
query99	1098	1132	990	990
Total cold run time: 254950 ms
Total hot run time: 171971 ms

@sollhui sollhui force-pushed the fix_stream_load_recorder_shutdown branch from eacdc0f to 563b1b4 Compare May 9, 2026 09:33
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 9, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-H: Total hot run time: 29761 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 563b1b44b60a1d0af4dcfac12b3d1b038ed616ef, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17634	4130	3920	3920
q2	q3	10720	898	607	607
q4	4660	461	345	345
q5	7473	1321	1138	1138
q6	194	173	145	145
q7	906	944	762	762
q8	9325	1397	1275	1275
q9	5669	5374	5356	5356
q10	6232	2089	1823	1823
q11	458	263	258	258
q12	634	419	297	297
q13	18180	3349	2729	2729
q14	295	284	268	268
q15	q16	898	852	785	785
q17	977	1048	832	832
q18	6396	5678	5562	5562
q19	1134	1230	1088	1088
q20	505	397	264	264
q21	4786	2391	1949	1949
q22	499	398	358	358
Total cold run time: 97575 ms
Total hot run time: 29761 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4791	4687	5039	4687
q2	q3	4666	4825	4198	4198
q4	2149	2187	1433	1433
q5	5048	5037	5300	5037
q6	210	175	145	145
q7	2110	1822	1642	1642
q8	3331	3124	3140	3124
q9	8446	8427	8479	8427
q10	4481	4530	4264	4264
q11	601	426	392	392
q12	699	747	530	530
q13	3228	3543	2947	2947
q14	303	310	293	293
q15	q16	779	786	690	690
q17	1450	1300	1245	1245
q18	7947	7204	7174	7174
q19	1175	1170	1218	1170
q20	2243	2279	1941	1941
q21	6191	5462	4929	4929
q22	550	529	420	420
Total cold run time: 60398 ms
Total hot run time: 54688 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-DS: Total hot run time: 170876 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 563b1b44b60a1d0af4dcfac12b3d1b038ed616ef, data reload: false

query5	4292	660	517	517
query6	328	223	203	203
query7	4205	556	314	314
query8	318	241	219	219
query9	8799	4033	4025	4025
query10	450	342	290	290
query11	5774	2367	2208	2208
query12	191	132	126	126
query13	1290	628	432	432
query14	5978	5376	5059	5059
query14_1	4385	4407	4400	4400
query15	217	209	188	188
query16	1039	439	445	439
query17	1163	794	653	653
query18	2482	511	366	366
query19	227	217	186	186
query20	142	132	135	132
query21	220	144	121	121
query22	13629	13942	14603	13942
query23	17347	16589	16203	16203
query23_1	16341	16260	16354	16260
query24	7527	1868	1369	1369
query24_1	1384	1342	1347	1342
query25	553	503	428	428
query26	1313	327	165	165
query27	2803	581	324	324
query28	4379	1938	1911	1911
query29	968	616	504	504
query30	296	232	200	200
query31	1093	1084	938	938
query32	79	71	73	71
query33	539	347	281	281
query34	1174	1094	631	631
query35	740	767	663	663
query36	1338	1322	1169	1169
query37	141	99	88	88
query38	3200	3135	3055	3055
query39	923	918	884	884
query39_1	865	863	882	863
query40	232	152	131	131
query41	64	64	57	57
query42	110	110	111	110
query43	323	322	276	276
query44	
query45	213	199	199	199
query46	1040	1177	745	745
query47	2292	2294	2174	2174
query48	402	410	308	308
query49	621	526	421	421
query50	690	285	236	236
query51	4336	4231	4178	4178
query52	104	102	92	92
query53	256	270	208	208
query54	306	294	248	248
query55	96	87	85	85
query56	295	290	299	290
query57	1392	1369	1293	1293
query58	291	267	292	267
query59	1575	1641	1416	1416
query60	368	336	327	327
query61	159	158	156	156
query62	670	618	543	543
query63	244	209	202	202
query64	2440	840	681	681
query65	
query66	1747	513	387	387
query67	30015	29977	29901	29901
query68	
query69	472	338	298	298
query70	977	961	982	961
query71	322	277	277	277
query72	3092	2734	2383	2383
query73	827	751	415	415
query74	5036	4907	4732	4732
query75	2775	2644	2330	2330
query76	2318	1098	758	758
query77	409	435	335	335
query78	12970	12925	12377	12377
query79	1457	1013	748	748
query80	679	581	494	494
query81	455	282	234	234
query82	1323	161	121	121
query83	351	282	240	240
query84	255	143	114	114
query85	867	517	485	485
query86	393	329	324	324
query87	3399	3338	3181	3181
query88	3512	2675	2632	2632
query89	434	382	337	337
query90	1886	180	186	180
query91	179	176	139	139
query92	80	81	74	74
query93	944	966	551	551
query94	529	367	298	298
query95	651	461	352	352
query96	1035	790	312	312
query97	2687	2701	2535	2535
query98	239	230	248	230
query99	1107	1143	979	979
Total cold run time: 252985 ms
Total hot run time: 170876 ms

@sollhui sollhui force-pushed the fix_stream_load_recorder_shutdown branch from 563b1b4 to 2208e5f Compare May 9, 2026 10:41
@sollhui
Copy link
Copy Markdown
Contributor Author

sollhui commented May 9, 2026

run buildall

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-H: Total hot run time: 29358 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 2208e5f94849eeb7ffedaa9fe49951ae375e9bce, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17611	3863	3854	3854
q2	q3	10718	861	597	597
q4	4660	449	345	345
q5	7433	1337	1161	1161
q6	185	171	134	134
q7	963	1038	753	753
q8	9969	1424	1294	1294
q9	5672	5317	5289	5289
q10	6305	2107	1772	1772
q11	472	257	258	257
q12	691	416	289	289
q13	18208	3350	2740	2740
q14	286	286	260	260
q15	q16	903	858	787	787
q17	1010	1074	721	721
q18	6427	5687	5600	5600
q19	1340	1240	1067	1067
q20	500	395	262	262
q21	4766	2267	1874	1874
q22	421	354	302	302
Total cold run time: 98540 ms
Total hot run time: 29358 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4144	4067	4053	4053
q2	q3	4642	4738	4179	4179
q4	2085	2172	1368	1368
q5	4940	4944	5212	4944
q6	195	164	132	132
q7	2020	1768	1936	1768
q8	3438	3183	3195	3183
q9	8447	8444	8409	8409
q10	4485	4470	4222	4222
q11	580	438	413	413
q12	711	753	530	530
q13	3212	3602	2956	2956
q14	299	303	272	272
q15	q16	760	779	725	725
q17	1341	1317	1259	1259
q18	8085	7115	7148	7115
q19	1175	1143	1127	1127
q20	2252	2261	1954	1954
q21	6156	5379	4824	4824
q22	549	514	415	415
Total cold run time: 59516 ms
Total hot run time: 53848 ms

@hello-stephen
Copy link
Copy Markdown
Contributor

hello-stephen commented May 9, 2026

TPC-DS: Total hot run time: 169822 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 2208e5f94849eeb7ffedaa9fe49951ae375e9bce, data reload: false

query5	4332	656	513	513
query6	334	216	199	199
query7	4249	591	314	314
query8	319	237	220	220
query9	8830	3991	3998	3991
query10	461	356	296	296
query11	5788	2440	2163	2163
query12	180	157	125	125
query13	1293	597	440	440
query14	6731	5329	5047	5047
query14_1	4360	4341	4362	4341
query15	214	199	184	184
query16	997	461	416	416
query17	1134	730	631	631
query18	2740	483	350	350
query19	217	201	171	171
query20	142	132	144	132
query21	220	144	123	123
query22	13641	13594	13349	13349
query23	17095	16338	16582	16338
query23_1	16254	16218	16284	16218
query24	7688	1822	1429	1429
query24_1	1381	1403	1454	1403
query25	597	552	494	494
query26	1356	326	179	179
query27	2969	613	356	356
query28	4435	1979	1989	1979
query29	1038	644	539	539
query30	305	234	200	200
query31	1128	1068	940	940
query32	90	77	73	73
query33	549	355	299	299
query34	1173	1184	654	654
query35	785	791	673	673
query36	1315	1317	1141	1141
query37	155	105	91	91
query38	3188	3163	3106	3106
query39	929	913	913	913
query39_1	870	876	873	873
query40	237	157	144	144
query41	69	64	64	64
query42	111	108	108	108
query43	324	323	327	323
query44	
query45	215	211	195	195
query46	1103	1179	706	706
query47	2264	2246	2161	2161
query48	382	409	295	295
query49	638	512	432	432
query50	721	287	209	209
query51	4320	4286	4180	4180
query52	102	104	95	95
query53	252	279	199	199
query54	312	266	254	254
query55	95	86	86	86
query56	304	308	307	307
query57	1396	1361	1300	1300
query58	288	262	260	260
query59	1548	1637	1441	1441
query60	349	324	326	324
query61	152	153	152	152
query62	673	612	556	556
query63	244	195	202	195
query64	2325	824	743	743
query65	
query66	1686	513	385	385
query67	29944	29313	29777	29313
query68	
query69	461	339	296	296
query70	1032	1024	1007	1007
query71	296	279	273	273
query72	2997	2722	2410	2410
query73	878	770	431	431
query74	5064	4895	4709	4709
query75	2772	2667	2326	2326
query76	2268	1131	747	747
query77	420	414	340	340
query78	12814	13010	12299	12299
query79	1512	938	711	711
query80	1388	580	490	490
query81	507	283	240	240
query82	1269	159	128	128
query83	359	275	251	251
query84	262	143	114	114
query85	924	513	446	446
query86	464	336	304	304
query87	3464	3354	3243	3243
query88	3518	2678	2603	2603
query89	450	383	336	336
query90	1926	182	170	170
query91	179	168	136	136
query92	75	77	71	71
query93	963	964	569	569
query94	725	356	290	290
query95	678	470	342	342
query96	983	731	362	362
query97	2682	2692	2575	2575
query98	241	232	228	228
query99	1096	1132	989	989
Total cold run time: 255010 ms
Total hot run time: 169822 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@liaoxin01 liaoxin01 liaoxin01 left review comments

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

dev/3.1.x dev/4.0.x dev/4.1.x reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sollhui @hello-stephen @liaoxin01