Skip to content

[opt](memory) reuse tablet schema in load path #58476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liaoxin01 merged 1 commit into from
Dec 9, 2025
Merged

[opt](memory) reuse tablet schema in load path #58476

liaoxin01 merged 1 commit into from
Dec 9, 2025
+20 −2

Conversation

@sollhui
Copy link
Contributor

@sollhui sollhui commented Nov 27, 2025 •
edited
Loading

What problem does this PR solve?

Background

build_current_tablet_schema may consume a significant amount of memory when load to a wide table with a large number of buckets:
image

Solution

Reuse tablet schema in load path to save load memory.

Test Result

Machine configuration: 16c 64G 3BE
Table Schema: 5000 column, 256 bucket

Each Be reduces 1G memory.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 27, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@sollhui sollhui force-pushed the reuse_tablet_schema branch from b17481c to 4876f7a Compare November 27, 2025 15:34
@sollhui
Copy link
Contributor Author

sollhui commented Nov 27, 2025

run buildall

1 similar comment
@sollhui
Copy link
Contributor Author

sollhui commented Nov 28, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

TPC-H: Total hot run time: 34123 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit e151f880d10da324eed136e6ed50ab96389e389f, data reload: false

------ Round 1 ----------------------------------
q1	17700	5157	4936	4936
q2	2024	314	203	203
q3	10283	1298	725	725
q4	10235	902	357	357
q5	7542	2462	2275	2275
q6	195	174	142	142
q7	948	766	648	648
q8	9361	1333	1022	1022
q9	6973	5292	5301	5292
q10	6873	2257	1808	1808
q11	521	300	291	291
q12	337	373	221	221
q13	17783	3635	3011	3011
q14	247	237	214	214
q15	577	527	523	523
q16	1045	1015	967	967
q17	597	782	485	485
q18	7360	6975	6926	6926
q19	1086	951	574	574
q20	348	345	227	227
q21	3547	3737	2323	2323
q22	1036	956	953	953
Total cold run time: 106618 ms
Total hot run time: 34123 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4919	4980	4966	4966
q2	343	403	312	312
q3	2166	2661	2273	2273
q4	1310	1779	1291	1291
q5	4199	4311	4487	4311
q6	213	177	132	132
q7	2018	2029	1857	1857
q8	2708	2412	2424	2412
q9	7536	7488	7476	7476
q10	3044	3298	2823	2823
q11	597	507	486	486
q12	698	805	632	632
q13	3543	3890	3311	3311
q14	306	290	294	290
q15	574	521	515	515
q16	1109	1108	1053	1053
q17	1139	1404	1333	1333
q18	8084	7533	7566	7533
q19	783	769	771	769
q20	2014	2082	1808	1808
q21	4585	4247	4153	4153
q22	1066	1040	972	972
Total cold run time: 52954 ms
Total hot run time: 50708 ms

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

TPC-DS: Total hot run time: 184639 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit e151f880d10da324eed136e6ed50ab96389e389f, data reload: false

query1	1049	407	391	391
query2	6582	1561	1607	1561
query3	6759	236	221	221
query4	24442	22949	22792	22792
query5	4925	621	484	484
query6	357	252	240	240
query7	4670	525	309	309
query8	317	264	254	254
query9	8736	2596	2582	2582
query10	537	362	317	317
query11	15137	15161	14886	14886
query12	197	122	117	117
query13	1692	571	455	455
query14	11441	8843	8887	8843
query15	210	203	192	192
query16	7692	722	520	520
query17	1238	778	654	654
query18	2052	440	351	351
query19	229	235	187	187
query20	129	127	121	121
query21	222	137	117	117
query22	4025	3910	3827	3827
query23	32659	32010	31896	31896
query24	8442	2415	2408	2408
query25	639	567	511	511
query26	1246	283	171	171
query27	2723	495	361	361
query28	4336	2139	2127	2127
query29	832	645	530	530
query30	316	241	212	212
query31	862	720	631	631
query32	92	76	76	76
query33	616	390	340	340
query34	865	872	542	542
query35	790	823	723	723
query36	889	900	828	828
query37	130	107	92	92
query38	3355	3337	3209	3209
query39	1481	1407	1417	1407
query40	225	129	118	118
query41	63	62	60	60
query42	123	108	111	108
query43	449	444	424	424
query44	1287	767	753	753
query45	203	189	185	185
query46	881	1001	640	640
query47	1706	1729	1628	1628
query48	390	420	354	354
query49	753	505	411	411
query50	674	683	410	410
query51	3919	3999	3837	3837
query52	110	109	112	109
query53	256	269	186	186
query54	315	287	266	266
query55	94	93	98	93
query56	362	335	327	327
query57	1113	1187	1077	1077
query58	286	276	274	274
query59	2343	2359	2419	2359
query60	355	360	346	346
query61	162	163	159	159
query62	779	741	651	651
query63	239	199	192	192
query64	4528	1216	885	885
query65	4082	3967	3963	3963
query66	1101	433	341	341
query67	15205	15079	14937	14937
query68	8506	948	614	614
query69	522	337	304	304
query70	1271	1265	1217	1217
query71	508	343	327	327
query72	5793	4818	4787	4787
query73	687	577	337	337
query74	8554	8898	8692	8692
query75	4029	3284	2842	2842
query76	3703	1127	706	706
query77	813	409	312	312
query78	9455	9687	8963	8963
query79	2026	850	585	585
query80	649	590	500	500
query81	495	276	232	232
query82	444	162	136	136
query83	267	267	257	257
query84	260	124	96	96
query85	942	489	437	437
query86	341	296	285	285
query87	3502	3545	3374	3374
query88	3846	2288	2321	2288
query89	398	342	300	300
query90	1978	239	211	211
query91	170	170	146	146
query92	84	76	71	71
query93	1279	1001	662	662
query94	745	467	338	338
query95	509	411	403	403
query96	517	557	288	288
query97	2931	2979	2898	2898
query98	238	212	213	212
query99	1322	1399	1241	1241
Total cold run time: 271698 ms
Total hot run time: 184639 ms

@doris-robot
Copy link

doris-robot commented Nov 28, 2025

ClickBench: Total hot run time: 27.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit e151f880d10da324eed136e6ed50ab96389e389f, data reload: false

query1	0.06	0.05	0.05
query2	0.09	0.05	0.05
query3	0.25	0.10	0.08
query4	1.60	0.12	0.11
query5	0.27	0.25	0.25
query6	1.19	0.64	0.64
query7	0.03	0.03	0.02
query8	0.05	0.04	0.05
query9	0.58	0.52	0.51
query10	0.56	0.56	0.55
query11	0.16	0.11	0.12
query12	0.14	0.13	0.11
query13	0.62	0.60	0.60
query14	0.99	0.99	0.99
query15	0.83	0.79	0.80
query16	0.42	0.40	0.42
query17	1.05	1.01	1.02
query18	0.23	0.21	0.22
query19	1.97	1.90	1.82
query20	0.02	0.01	0.01
query21	15.49	0.27	0.16
query22	5.01	0.05	0.05
query23	16.18	0.28	0.10
query24	1.14	0.39	0.56
query25	0.11	0.05	0.08
query26	0.15	0.13	0.13
query27	0.09	0.05	0.05
query28	4.52	1.19	1.02
query29	12.55	3.83	3.32
query30	0.28	0.15	0.12
query31	2.82	0.60	0.39
query32	3.23	0.54	0.46
query33	3.08	3.01	3.09
query34	16.88	5.12	4.49
query35	4.52	4.51	4.52
query36	0.67	0.49	0.49
query37	0.11	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.13	0.14
query41	0.08	0.04	0.03
query42	0.05	0.03	0.04
query43	0.04	0.04	0.04
Total cold run time: 98.4 s
Total hot run time: 27.45 s

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 28, 2025

BE UT Coverage Report

Increment line coverage 14.63% (6/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 52.88% (18503/34988)
Line Coverage 38.42% (169487/441090)
Region Coverage 33.23% (131590/395974)
Branch Coverage 34.18% (56662/165779)

@hello-stephen
Copy link
Contributor

hello-stephen commented Nov 28, 2025

BE Regression && UT Coverage Report

Increment line coverage 82.93% (34/41) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 71.68% (24589/34306)
Line Coverage 58.22% (256588/440696)
Region Coverage 53.28% (213534/400784)
Branch Coverage 54.79% (91325/166674)

@sollhui sollhui force-pushed the reuse_tablet_schema branch from e151f88 to 01dc885 Compare December 2, 2025 06:49
@sollhui
Copy link
Contributor Author

sollhui commented Dec 3, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-H: Total hot run time: 34624 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 01dc8859f2c281a963c60737b1d3d01cb88d4c0f, data reload: false

------ Round 1 ----------------------------------
q1	17671	5118	4887	4887
q2	2068	325	215	215
q3	10251	1342	755	755
q4	10237	913	328	328
q5	7574	2488	2206	2206
q6	188	176	139	139
q7	969	795	643	643
q8	9366	1620	1172	1172
q9	7085	5367	5381	5367
q10	6876	2205	1792	1792
q11	542	299	289	289
q12	341	372	223	223
q13	17794	3714	3049	3049
q14	238	252	219	219
q15	591	546	517	517
q16	918	881	814	814
q17	690	810	526	526
q18	7721	7161	7168	7161
q19	1091	986	613	613
q20	364	361	233	233
q21	4055	3589	2501	2501
q22	1044	997	975	975
Total cold run time: 107674 ms
Total hot run time: 34624 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4978	4925	4967	4925
q2	321	370	327	327
q3	2135	2696	2279	2279
q4	1309	1790	1300	1300
q5	4241	4607	4603	4603
q6	244	181	134	134
q7	2102	2034	1840	1840
q8	2991	2636	2523	2523
q9	7540	7438	7609	7438
q10	3049	3327	2899	2899
q11	602	512	487	487
q12	686	770	648	648
q13	3572	4012	3289	3289
q14	296	324	284	284
q15	554	516	499	499
q16	921	931	889	889
q17	1289	1422	1402	1402
q18	7856	7697	7651	7651
q19	891	935	869	869
q20	1996	2094	1876	1876
q21	4678	4250	4133	4133
q22	1067	1032	991	991
Total cold run time: 53318 ms
Total hot run time: 51286 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

TPC-DS: Total hot run time: 182664 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 01dc8859f2c281a963c60737b1d3d01cb88d4c0f, data reload: false

query1	1107	420	395	395
query2	6605	1194	1182	1182
query3	6758	226	228	226
query4	25615	23315	22991	22991
query5	5541	668	504	504
query6	343	254	228	228
query7	4661	532	321	321
query8	331	258	259	258
query9	8749	2634	2666	2634
query10	564	380	319	319
query11	15444	14815	15313	14815
query12	186	124	120	120
query13	1701	594	455	455
query14	9639	6041	6080	6041
query15	226	200	195	195
query16	7638	708	541	541
query17	1381	775	613	613
query18	2043	427	327	327
query19	216	197	192	192
query20	129	135	122	122
query21	223	140	114	114
query22	3843	3931	3858	3858
query23	32943	31885	32031	31885
query24	8482	2463	2409	2409
query25	604	514	448	448
query26	1246	278	175	175
query27	2735	509	356	356
query28	4359	2175	2169	2169
query29	830	666	530	530
query30	323	247	210	210
query31	818	771	651	651
query32	90	82	83	82
query33	612	387	336	336
query34	832	893	538	538
query35	790	842	734	734
query36	904	945	855	855
query37	135	113	87	87
query38	3829	3881	3859	3859
query39	1473	1405	1429	1405
query40	232	134	122	122
query41	68	64	63	63
query42	128	119	119	119
query43	445	478	435	435
query44	1328	770	763	763
query45	203	193	188	188
query46	902	1037	656	656
query47	1690	1722	1690	1690
query48	412	452	323	323
query49	802	520	409	409
query50	704	704	412	412
query51	3913	3896	3846	3846
query52	115	114	104	104
query53	246	265	200	200
query54	314	306	296	296
query55	97	95	101	95
query56	328	352	351	351
query57	1151	1175	1072	1072
query58	295	282	314	282
query59	2324	2425	2249	2249
query60	370	356	331	331
query61	167	165	168	165
query62	785	725	644	644
query63	239	200	199	199
query64	4443	1228	920	920
query65	4067	3979	3981	3979
query66	1062	449	344	344
query67	15494	15118	14722	14722
query68	4884	1019	654	654
query69	524	349	310	310
query70	1102	992	1019	992
query71	431	336	319	319
query72	5822	5174	5086	5086
query73	676	582	355	355
query74	8635	8801	8741	8741
query75	3077	3061	2560	2560
query76	3231	1219	772	772
query77	508	431	330	330
query78	9647	9950	8917	8917
query79	1887	844	593	593
query80	952	597	518	518
query81	513	270	249	249
query82	443	140	112	112
query83	271	261	250	250
query84	258	122	105	105
query85	906	494	456	456
query86	336	324	290	290
query87	4170	4077	4003	4003
query88	3070	2361	2345	2345
query89	396	343	290	290
query90	1771	231	232	231
query91	179	175	144	144
query92	76	73	69	69
query93	1256	1076	682	682
query94	740	453	350	350
query95	509	433	419	419
query96	550	552	290	290
query97	2614	2717	2548	2548
query98	241	228	218	218
query99	1336	1380	1307	1307
Total cold run time: 267283 ms
Total hot run time: 182664 ms

@doris-robot
Copy link

doris-robot commented Dec 3, 2025

ClickBench: Total hot run time: 28.08 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 01dc8859f2c281a963c60737b1d3d01cb88d4c0f, data reload: false

query1	0.06	0.05	0.05
query2	0.11	0.06	0.05
query3	0.25	0.08	0.08
query4	1.62	0.11	0.12
query5	0.28	0.27	0.27
query6	1.18	0.65	0.63
query7	0.04	0.03	0.03
query8	0.06	0.04	0.04
query9	0.56	0.52	0.51
query10	0.55	0.55	0.56
query11	0.16	0.10	0.11
query12	0.15	0.12	0.12
query13	0.62	0.60	0.61
query14	1.00	0.99	1.02
query15	0.83	0.82	0.81
query16	0.42	0.40	0.42
query17	1.06	1.04	1.06
query18	0.24	0.22	0.22
query19	1.95	1.89	1.89
query20	0.02	0.01	0.01
query21	15.47	0.29	0.13
query22	4.98	0.05	0.05
query23	16.09	0.28	0.10
query24	1.34	0.70	0.83
query25	0.08	0.08	0.14
query26	0.14	0.14	0.13
query27	0.08	0.05	0.06
query28	5.12	1.21	1.03
query29	12.59	4.10	3.38
query30	0.28	0.14	0.12
query31	2.83	0.64	0.41
query32	3.23	0.57	0.48
query33	3.23	3.02	3.11
query34	17.02	5.14	4.55
query35	4.55	4.58	4.49
query36	0.65	0.50	0.50
query37	0.11	0.06	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.03
query40	0.17	0.13	0.13
query41	0.09	0.04	0.03
query42	0.05	0.04	0.03
query43	0.04	0.04	0.04
Total cold run time: 99.42 s
Total hot run time: 28.08 s

@sollhui sollhui force-pushed the reuse_tablet_schema branch from e5cd56f to 7424a3d Compare December 4, 2025 10:08
@sollhui sollhui changed the title [draft] resue tablet schema in load level [opt](memory) reuse tablet schema in load path Dec 4, 2025
@sollhui
Copy link
Contributor Author

sollhui commented Dec 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

TPC-H: Total hot run time: 34424 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7424a3dab1d1bbb9a0bb866a1f38867e0dd03edb, data reload: false

------ Round 1 ----------------------------------
q1	17668	5088	4877	4877
q2	2038	298	186	186
q3	10307	1311	728	728
q4	10226	859	316	316
q5	7525	2375	2205	2205
q6	184	169	136	136
q7	955	781	646	646
q8	9372	1415	1088	1088
q9	6847	5264	5312	5264
q10	6834	2211	1788	1788
q11	538	307	286	286
q12	344	370	217	217
q13	17804	3736	2997	2997
q14	228	238	218	218
q15	601	529	527	527
q16	932	914	831	831
q17	695	791	530	530
q18	7896	7232	7262	7232
q19	1088	983	611	611
q20	390	347	222	222
q21	4108	3655	2540	2540
q22	1059	1038	979	979
Total cold run time: 107639 ms
Total hot run time: 34424 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4995	4918	4958	4918
q2	332	416	335	335
q3	2152	2678	2314	2314
q4	1320	1759	1292	1292
q5	4211	4525	4595	4525
q6	224	179	137	137
q7	2013	1957	1827	1827
q8	2586	2549	2462	2462
q9	7597	7479	7634	7479
q10	3086	3240	2862	2862
q11	585	509	491	491
q12	678	774	610	610
q13	3459	3978	3300	3300
q14	291	328	290	290
q15	568	518	509	509
q16	889	914	909	909
q17	1212	1526	1554	1526
q18	7824	7576	7591	7576
q19	882	791	819	791
q20	1891	1944	1804	1804
q21	4596	4251	4125	4125
q22	1103	1046	990	990
Total cold run time: 52494 ms
Total hot run time: 51072 ms

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

TPC-DS: Total hot run time: 179848 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7424a3dab1d1bbb9a0bb866a1f38867e0dd03edb, data reload: false

query5	5105	673	497	497
query6	338	244	213	213
query7	4662	482	274	274
query8	303	245	243	243
query9	8759	2646	2660	2646
query10	562	329	278	278
query11	15278	14798	14559	14559
query12	180	126	117	117
query13	1699	507	378	378
query14	6191	3354	3050	3050
query14_1	2949	2888	2919	2888
query15	214	197	185	185
query16	7700	483	457	457
query17	1224	736	607	607
query18	2048	436	345	345
query19	221	207	178	178
query20	135	121	121	121
query21	220	149	121	121
query22	3975	4081	3962	3962
query23	16716	16303	16002	16002
query23_1	16085	16060	16087	16060
query24	7345	1658	1228	1228
query24_1	1197	1223	1228	1223
query25	631	503	445	445
query26	1257	286	181	181
query27	2897	471	302	302
query28	4379	2190	2168	2168
query29	839	594	488	488
query30	321	248	219	219
query31	834	699	636	636
query32	90	76	74	74
query33	685	354	315	315
query34	868	973	544	544
query35	796	810	730	730
query36	908	906	830	830
query37	125	94	76	76
query38	3905	3880	3825	3825
query39	747	734	720	720
query39_1	697	704	695	695
query40	239	131	116	116
query41	69	67	62	62
query42	127	101	97	97
query43	456	420	387	387
query44	1299	757	756	756
query45	202	188	181	181
query46	914	971	596	596
query47	1674	1738	1650	1650
query48	413	318	233	233
query49	760	431	388	388
query50	700	304	238	238
query51	3816	3962	3779	3779
query52	113	92	88	88
query53	235	231	174	174
query54	322	270	252	252
query55	93	77	78	77
query56	348	285	291	285
query57	1160	1151	1082	1082
query58	299	297	255	255
query59	2368	2481	2318	2318
query60	353	323	300	300
query61	159	150	158	150
query62	794	695	622	622
query63	234	176	179	176
query64	4406	1157	878	878
query65	4044	3943	3955	3943
query66	1216	456	351	351
query67	15285	15062	14781	14781
query68	8518	940	688	688
query69	544	297	268	268
query70	1134	956	964	956
query71	468	303	272	272
query72	5941	4902	4898	4898
query73	683	544	299	299
query74	8806	8733	8495	8495
query75	3268	3028	2495	2495
query76	3450	1148	754	754
query77	579	416	298	298
query78	9487	9574	8951	8951
query79	1886	846	590	590
query80	686	535	471	471
query81	492	271	240	240
query82	482	132	105	105
query83	283	271	254	254
query84	260	113	93	93
query85	946	489	458	458
query86	392	312	283	283
query87	4073	4006	4012	4006
query88	4270	2131	2139	2131
query89	398	317	284	284
query90	2039	164	156	156
query91	178	167	141	141
query92	81	65	68	65
query93	2007	1036	675	675
query94	771	304	287	287
query95	568	328	384	328
query96	529	504	217	217
query97	2644	2702	2577	2577
query98	251	201	192	192
query99	1370	1323	1242	1242
Total cold run time: 272974 ms
Total hot run time: 179848 ms

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

ClickBench: Total hot run time: 27.35 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7424a3dab1d1bbb9a0bb866a1f38867e0dd03edb, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.05	0.05
query3	0.25	0.09	0.09
query4	1.61	0.11	0.12
query5	0.29	0.27	0.26
query6	1.16	0.65	0.63
query7	0.03	0.02	0.02
query8	0.05	0.04	0.05
query9	0.57	0.51	0.50
query10	0.56	0.57	0.55
query11	0.16	0.10	0.11
query12	0.15	0.11	0.10
query13	0.63	0.60	0.59
query14	1.01	1.00	0.99
query15	0.81	0.81	0.81
query16	0.41	0.42	0.42
query17	1.02	1.07	1.03
query18	0.22	0.21	0.21
query19	1.97	1.74	1.86
query20	0.01	0.01	0.01
query21	15.43	0.28	0.14
query22	4.55	0.06	0.05
query23	15.98	0.27	0.10
query24	1.89	0.25	0.60
query25	0.09	0.07	0.06
query26	0.14	0.14	0.13
query27	0.05	0.05	0.05
query28	4.55	1.23	1.02
query29	12.57	4.00	3.24
query30	0.27	0.14	0.11
query31	2.83	0.63	0.40
query32	3.22	0.56	0.47
query33	3.02	3.09	3.14
query34	16.88	5.14	4.57
query35	4.58	4.61	4.54
query36	0.65	0.50	0.48
query37	0.11	0.06	0.06
query38	0.07	0.05	0.04
query39	0.04	0.03	0.03
query40	0.18	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 98.34 s
Total hot run time: 27.35 s

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.34% (18632/34931)
Line Coverage 39.00% (172066/441206)
Region Coverage 33.61% (133238/396420)
Branch Coverage 34.57% (57342/165883)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.22% (24736/34253)
Line Coverage 58.91% (259682/440827)
Region Coverage 53.79% (215813/401240)
Branch Coverage 55.38% (92370/166782)

@sollhui sollhui force-pushed the reuse_tablet_schema branch from 7424a3d to 0a2feda Compare December 4, 2025 13:13
@sollhui
Copy link
Contributor Author

sollhui commented Dec 4, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

TPC-H: Total hot run time: 34832 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0a2feda16afd34aafcefd620e8e3513b84d75753, data reload: false

------ Round 1 ----------------------------------
q1	17632	5226	5017	5017
q2	2074	320	200	200
q3	10233	1301	754	754
q4	10229	879	319	319
q5	7521	2461	2185	2185
q6	204	178	143	143
q7	976	799	659	659
q8	9367	1450	1126	1126
q9	7211	5356	5374	5356
q10	6871	2197	1788	1788
q11	521	310	298	298
q12	383	375	225	225
q13	17803	3698	3029	3029
q14	239	237	230	230
q15	596	531	523	523
q16	894	879	820	820
q17	667	787	596	596
q18	7394	7153	7126	7126
q19	1101	969	638	638
q20	393	344	227	227
q21	4123	3762	2625	2625
q22	1081	971	948	948
Total cold run time: 107513 ms
Total hot run time: 34832 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5044	5032	5048	5032
q2	345	411	301	301
q3	2163	2727	2339	2339
q4	1373	1787	1319	1319
q5	4384	4631	4446	4446
q6	214	188	141	141
q7	2043	2016	1832	1832
q8	2755	2645	2600	2600
q9	7613	7474	7498	7474
q10	3059	3282	2892	2892
q11	648	533	499	499
q12	669	743	622	622
q13	3711	3936	3353	3353
q14	300	322	284	284
q15	564	537	523	523
q16	904	915	902	902
q17	1243	1571	1440	1440
q18	8043	7675	7765	7675
q19	937	925	916	916
q20	2014	2020	1819	1819
q21	4683	4353	4257	4257
q22	1097	1030	982	982
Total cold run time: 53806 ms
Total hot run time: 51648 ms

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

TPC-DS: Total hot run time: 180432 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0a2feda16afd34aafcefd620e8e3513b84d75753, data reload: false

query5	4872	659	494	494
query6	366	227	209	209
query7	4664	486	294	294
query8	302	245	244	244
query9	8729	2653	2672	2653
query10	575	318	280	280
query11	15282	14840	14866	14840
query12	181	117	117	117
query13	1706	502	380	380
query14	6001	3303	3053	3053
query14_1	2979	2895	2928	2895
query15	208	199	183	183
query16	7501	503	467	467
query17	1221	737	613	613
query18	2031	448	370	370
query19	227	188	162	162
query20	132	118	119	118
query21	215	134	119	119
query22	3866	4079	3830	3830
query23	17142	16237	15918	15918
query23_1	16015	16051	16075	16051
query24	7066	1623	1215	1215
query24_1	1220	1201	1251	1201
query25	608	471	420	420
query26	946	281	177	177
query27	2891	475	309	309
query28	4395	2187	2179	2179
query29	757	572	450	450
query30	324	244	220	220
query31	821	691	643	643
query32	80	68	68	68
query33	665	354	290	290
query34	869	894	544	544
query35	796	829	747	747
query36	903	941	822	822
query37	122	97	82	82
query38	3922	3905	3879	3879
query39	761	741	743	741
query39_1	691	703	703	703
query40	223	136	122	122
query41	71	64	62	62
query42	124	99	100	99
query43	452	422	394	394
query44	1321	774	770	770
query45	200	193	182	182
query46	908	964	597	597
query47	1665	1709	1630	1630
query48	404	325	234	234
query49	759	449	359	359
query50	687	304	235	235
query51	3831	3933	3885	3885
query52	123	95	84	84
query53	237	261	182	182
query54	332	266	242	242
query55	93	83	74	74
query56	342	288	290	288
query57	1163	1157	1098	1098
query58	293	262	257	257
query59	2277	2358	2327	2327
query60	356	318	299	299
query61	160	157	157	157
query62	772	688	630	630
query63	239	179	197	179
query64	3874	1176	900	900
query65	4075	3973	3995	3973
query66	1071	453	341	341
query67	14870	14728	14718	14718
query68	4644	962	682	682
query69	531	307	266	266
query70	1117	1019	960	960
query71	423	300	280	280
query72	6275	5196	5416	5196
query73	769	659	302	302
query74	8779	8806	8635	8635
query75	3079	3045	2532	2532
query76	3211	1171	749	749
query77	506	409	299	299
query78	9458	9603	8931	8931
query79	1773	863	580	580
query80	1712	590	465	465
query81	577	274	243	243
query82	432	130	102	102
query83	378	272	258	258
query84	266	116	100	100
query85	936	487	450	450
query86	470	323	308	308
query87	4009	4065	3951	3951
query88	3019	2120	2119	2119
query89	395	318	283	283
query90	1759	163	159	159
query91	173	164	162	162
query92	74	71	67	67
query93	1318	1035	684	684
query94	784	331	266	266
query95	564	385	328	328
query96	534	496	217	217
query97	2627	2673	2597	2597
query98	242	205	197	197
query99	1340	1305	1246	1246
Total cold run time: 265184 ms
Total hot run time: 180432 ms

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

ClickBench: Total hot run time: 27.59 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0a2feda16afd34aafcefd620e8e3513b84d75753, data reload: false

query1	0.05	0.04	0.04
query2	0.10	0.05	0.05
query3	0.26	0.09	0.09
query4	1.61	0.12	0.12
query5	0.26	0.26	0.25
query6	1.19	0.65	0.63
query7	0.03	0.02	0.03
query8	0.05	0.04	0.04
query9	0.58	0.50	0.49
query10	0.55	0.56	0.55
query11	0.16	0.11	0.11
query12	0.15	0.12	0.11
query13	0.63	0.63	0.60
query14	1.00	1.01	0.98
query15	0.82	0.80	0.81
query16	0.41	0.40	0.40
query17	1.01	0.99	1.01
query18	0.23	0.21	0.22
query19	1.82	1.80	1.85
query20	0.02	0.01	0.01
query21	15.43	0.30	0.15
query22	4.89	0.05	0.04
query23	15.89	0.29	0.10
query24	1.41	0.48	0.68
query25	0.09	0.07	0.05
query26	0.14	0.13	0.13
query27	0.08	0.07	0.05
query28	5.18	1.23	1.02
query29	12.59	4.06	3.24
query30	0.27	0.15	0.12
query31	2.81	0.62	0.41
query32	3.22	0.56	0.46
query33	3.06	3.05	3.10
query34	16.92	5.22	4.61
query35	4.52	4.60	4.55
query36	0.68	0.50	0.49
query37	0.11	0.06	0.06
query38	0.07	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.15	0.14
query41	0.09	0.03	0.04
query42	0.05	0.03	0.03
query43	0.04	0.03	0.03
Total cold run time: 98.68 s
Total hot run time: 27.59 s

@doris-robot
Copy link

doris-robot commented Dec 4, 2025

BE UT Coverage Report

Increment line coverage 0.00% (0/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 53.34% (18634/34932)
Line Coverage 39.01% (172117/441212)
Region Coverage 33.61% (133250/396421)
Branch Coverage 34.59% (57376/165883)

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 4, 2025

BE Regression && UT Coverage Report

Increment line coverage 100.00% (16/16) 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 72.16% (24719/34254)
Line Coverage 58.86% (259487/440833)
Region Coverage 53.85% (216049/401241)
Branch Coverage 55.32% (92269/166782)

@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2025

PR approved by anyone and no changes requested.

@sollhui sollhui mentioned this pull request Dec 8, 2025
Open
6 tasks
dataroaring
dataroaring approved these changes Dec 9, 2025
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Dec 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Dec 9, 2025

PR approved by at least one committer and no changes requested.

eldenmoon
eldenmoon approved these changes Dec 9, 2025
View reviewed changes
Copy link
Member

@eldenmoon eldenmoon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@liaoxin01 liaoxin01 merged commit e25dc87 into apache:master Dec 9, 2025
33 of 35 checks passed
github-actions bot pushed a commit that referenced this pull request Dec 9, 2025
@sollhui
[opt](memory) reuse tablet schema in load path (#58476)
c85d7da 
### What problem does this PR solve?

#### Background
`build_current_tablet_schema` may consume a significant amount of memory
when load to a wide table with a large number of buckets:
<img width="716" height="718" alt="image"
src="https://github.com/user-attachments/assets/0b08c807-39df-4ab5-b7af-5896e1265009"
/>

#### Solution
Reuse tablet schema in load path to save load memory.

#### Test Result
Machine configuration: 16c 64G 3BE
Table Schema: 5000 column, 256 bucket

Each Be reduces 1G memory.
@github-actions github-actions bot mentioned this pull request Dec 9, 2025
Merged
nagisa-kunhah pushed a commit to nagisa-kunhah/doris that referenced this pull request Dec 14, 2025
@sollhui @nagisa-kunhah
[opt](memory) reuse tablet schema in load path (apache#58476)
5c3d72b 
### What problem does this PR solve?

#### Background
`build_current_tablet_schema` may consume a significant amount of memory
when load to a wide table with a large number of buckets:
<img width="716" height="718" alt="image"
src="https://github.com/user-attachments/assets/0b08c807-39df-4ab5-b7af-5896e1265009"
/>

#### Solution
Reuse tablet schema in load path to save load memory.

#### Test Result
Machine configuration: 16c 64G 3BE
Table Schema: 5000 column, 256 bucket

Each Be reduces 1G memory.
yiguolei pushed a commit that referenced this pull request Jan 5, 2026
@github-actions @sollhui
branch-4.0: [opt](memory) reuse tablet schema in load path #58476 (#5…
6e38e04 
…8847)

Cherry-picked from #58476

Co-authored-by: hui lai <laihui@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eldenmoon eldenmoon eldenmoon approved these changes

@dataroaring dataroaring dataroaring approved these changes

+1 more reviewer

@0AyanamiRei 0AyanamiRei 0AyanamiRei approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.x dev/3.1.x-conflict dev/4.0.3-merged reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

8 participants

@sollhui @hello-stephen @doris-robot @eldenmoon @dataroaring @0AyanamiRei @yiguolei @liaoxin01