Skip to content

[Optimize](TabletMeta) Reuse TabletSchema in TabletMeta#46572

Merged
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:opt-tablet-meta
Jan 9, 2025
Merged

[Optimize](TabletMeta) Reuse TabletSchema in TabletMeta#46572
eldenmoon merged 1 commit intoapache:masterfrom
eldenmoon:opt-tablet-meta

Conversation

@eldenmoon
Copy link
Member

@eldenmoon eldenmoon commented Jan 7, 2025

In clusters with a large number of tablets, the sheer volume of TabletSchema instances can consume significant memory. This change introduces TabletSchema reuse in TabletMeta to optimize memory usage.

With large number of tablets:
56G memory consumption reduced to 4G

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

In clusters with a large number of tablets, the sheer volume of TabletSchema instances can consume significant memory. This change introduces TabletSchema reuse in TabletMeta to optimize memory usage.
@Thearas
Copy link
Contributor

Thearas commented Jan 7, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@eldenmoon
Copy link
Member Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33584 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c87311d51feda32403b942e9a9639dd033844e77, data reload: false

------ Round 1 ----------------------------------
q1	17593	6248	6176	6176
q2	2044	323	180	180
q3	10391	1505	745	745
q4	10212	917	444	444
q5	7512	2316	2175	2175
q6	217	185	146	146
q7	943	769	615	615
q8	9246	1517	1289	1289
q9	5508	5099	5133	5099
q10	6763	2348	1903	1903
q11	499	292	265	265
q12	365	381	218	218
q13	17784	3744	3062	3062
q14	239	238	225	225
q15	560	517	493	493
q16	631	635	589	589
q17	595	926	333	333
q18	7195	6656	6477	6477
q19	1461	1057	603	603
q20	324	329	186	186
q21	3078	2316	2053	2053
q22	364	336	308	308
Total cold run time: 103524 ms
Total hot run time: 33584 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6208	6429	6396	6396
q2	256	337	236	236
q3	2308	2683	2328	2328
q4	1442	1868	1386	1386
q5	4431	4900	5122	4900
q6	207	184	151	151
q7	2128	1988	1814	1814
q8	2744	2929	2820	2820
q9	7303	7190	7288	7190
q10	3100	3360	2805	2805
q11	592	520	502	502
q12	692	771	595	595
q13	3567	3917	3326	3326
q14	297	315	290	290
q15	581	496	511	496
q16	660	677	636	636
q17	1307	1830	1289	1289
q18	7695	7348	7047	7047
q19	870	1242	1187	1187
q20	1982	2021	1811	1811
q21	5607	5134	5070	5070
q22	655	648	608	608
Total cold run time: 54632 ms
Total hot run time: 52883 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 191036 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c87311d51feda32403b942e9a9639dd033844e77, data reload: false

query1	972	392	373	373
query2	6524	2307	2365	2307
query3	6715	216	210	210
query4	33760	23835	23945	23835
query5	4292	611	451	451
query6	279	205	208	205
query7	4647	499	301	301
query8	289	231	222	222
query9	9667	2679	2652	2652
query10	460	318	240	240
query11	18216	15344	15226	15226
query12	152	105	106	105
query13	1654	570	408	408
query14	10044	7433	7469	7433
query15	266	205	185	185
query16	8328	618	442	442
query17	1628	766	571	571
query18	2147	410	295	295
query19	223	182	157	157
query20	118	123	120	120
query21	212	127	110	110
query22	4315	4370	4414	4370
query23	34690	34536	33570	33570
query24	6485	2399	2247	2247
query25	450	452	374	374
query26	1048	250	154	154
query27	2054	456	332	332
query28	5341	2462	2412	2412
query29	521	534	401	401
query30	225	179	154	154
query31	971	905	821	821
query32	86	63	58	58
query33	504	345	297	297
query34	748	826	504	504
query35	789	815	740	740
query36	1025	1035	956	956
query37	116	92	77	77
query38	4065	4245	3973	3973
query39	1489	1414	1434	1414
query40	203	113	101	101
query41	46	47	44	44
query42	126	101	105	101
query43	527	516	493	493
query44	1295	791	800	791
query45	181	172	176	172
query46	854	1040	641	641
query47	1969	1937	1881	1881
query48	377	408	318	318
query49	763	475	386	386
query50	609	655	375	375
query51	7067	7075	6890	6890
query52	105	98	86	86
query53	220	248	181	181
query54	474	474	401	401
query55	83	82	78	78
query56	254	247	239	239
query57	1196	1151	1127	1127
query58	238	226	228	226
query59	2933	3077	2862	2862
query60	267	258	271	258
query61	134	117	108	108
query62	882	788	740	740
query63	226	190	195	190
query64	4493	987	663	663
query65	3285	3195	3201	3195
query66	1117	435	310	310
query67	16105	15778	15624	15624
query68	8707	688	517	517
query69	459	297	253	253
query70	1225	1160	1117	1117
query71	455	285	260	260
query72	6173	3837	3886	3837
query73	669	749	351	351
query74	9721	9292	8946	8946
query75	4348	3180	2620	2620
query76	4120	1172	769	769
query77	777	364	282	282
query78	10079	10145	9521	9521
query79	3430	816	600	600
query80	703	540	494	494
query81	476	271	231	231
query82	597	151	124	124
query83	198	166	140	140
query84	281	84	75	75
query85	789	375	312	312
query86	359	312	290	290
query87	4434	4593	4389	4389
query88	4180	2189	2218	2189
query89	424	337	296	296
query90	1911	195	186	186
query91	133	131	119	119
query92	76	57	66	57
query93	1662	854	524	524
query94	657	398	285	285
query95	333	258	256	256
query96	483	605	286	286
query97	2939	3052	2906	2906
query98	226	209	204	204
query99	1644	1549	1417	1417
Total cold run time: 294212 ms
Total hot run time: 191036 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.38 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit c87311d51feda32403b942e9a9639dd033844e77, data reload: false

query1	0.04	0.05	0.03
query2	0.06	0.04	0.03
query3	0.23	0.08	0.06
query4	1.61	0.11	0.10
query5	0.43	0.41	0.39
query6	1.18	0.66	0.64
query7	0.02	0.01	0.02
query8	0.04	0.04	0.03
query9	0.58	0.50	0.50
query10	0.56	0.55	0.55
query11	0.14	0.10	0.11
query12	0.15	0.10	0.11
query13	0.60	0.60	0.60
query14	2.71	2.73	2.71
query15	0.92	0.82	0.83
query16	0.39	0.39	0.37
query17	1.07	1.04	1.06
query18	0.23	0.20	0.21
query19	1.85	1.88	1.98
query20	0.01	0.01	0.01
query21	15.36	0.91	0.57
query22	0.76	0.75	0.67
query23	15.35	1.47	0.51
query24	2.93	1.11	1.32
query25	0.15	0.18	0.25
query26	0.20	0.13	0.13
query27	0.07	0.04	0.04
query28	14.34	1.52	1.04
query29	12.55	3.84	3.22
query30	0.24	0.09	0.07
query31	2.81	0.58	0.37
query32	3.24	0.55	0.46
query33	3.06	3.14	3.23
query34	16.82	5.03	4.50
query35	4.49	4.43	4.45
query36	0.65	0.52	0.48
query37	0.09	0.06	0.05
query38	0.04	0.04	0.03
query39	0.04	0.02	0.02
query40	0.17	0.13	0.12
query41	0.08	0.02	0.02
query42	0.04	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.34 s
Total hot run time: 31.38 s

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 38.86% (10128/26064)
Line Coverage: 29.90% (85687/286568)
Region Coverage: 29.00% (43725/150777)
Branch Coverage: 25.55% (22325/87382)
Coverage Report: http://coverage.selectdb-in.cc/coverage/c87311d51feda32403b942e9a9639dd033844e77_c87311d51feda32403b942e9a9639dd033844e77/report/index.html

@eldenmoon eldenmoon requested a review from dataroaring January 9, 2025 03:43
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Jan 9, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 9, 2025

PR approved by anyone and no changes requested.

Copy link
Contributor

@csun5285 csun5285 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@eldenmoon eldenmoon merged commit 7fc776f into apache:master Jan 9, 2025
18 of 20 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 9, 2025
In clusters with a large number of tablets, the sheer volume of
TabletSchema instances can consume significant memory. This change
introduces TabletSchema reuse in TabletMeta to optimize memory usage.

With large number of tablets:
56G memory consumption reduced to 4G
github-actions bot pushed a commit that referenced this pull request Jan 9, 2025
In clusters with a large number of tablets, the sheer volume of
TabletSchema instances can consume significant memory. This change
introduces TabletSchema reuse in TabletMeta to optimize memory usage.

With large number of tablets:
56G memory consumption reduced to 4G
yiguolei pushed a commit that referenced this pull request Jan 9, 2025
…46572 (#46687)

Cherry-picked from #46572

Co-authored-by: lihangyu <lihangyu@selectdb.com>
dataroaring pushed a commit that referenced this pull request Jan 13, 2025
…46572 (#46686)

Cherry-picked from #46572

Co-authored-by: lihangyu <lihangyu@selectdb.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/2.1.8-merged dev/3.0.4-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants