Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fix](tablet-meta) limit the data size of tablet meta #39455

Merged
merged 5 commits into from
Aug 26, 2024
Merged

[Fix](tablet-meta) limit the data size of tablet meta #39455

merged 5 commits into from
Aug 26, 2024

Conversation

luwei16
Copy link
Contributor

@luwei16 luwei16 commented Aug 16, 2024

No description provided.

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@luwei16 luwei16 changed the title [Fix](tablet-meta)limiting the size of tablet meta [Fix](tablet-meta) limiting the size of tablet meta Aug 16, 2024
@luwei16 luwei16 changed the title [Fix](tablet-meta) limiting the size of tablet meta [Fix](tablet-meta) limit the data size of tablet meta Aug 16, 2024
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 23, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 37994 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 4821ff3de3f7a150ad07611d03cb512254251e7b, data reload: false

------ Round 1 ----------------------------------
q1	18135	4531	4318	4318
q2	2032	185	177	177
q3	11795	985	1166	985
q4	10449	733	703	703
q5	7740	2891	2868	2868
q6	229	136	135	135
q7	988	600	577	577
q8	9341	2102	2094	2094
q9	7310	6532	6558	6532
q10	7006	2164	2203	2164
q11	447	247	243	243
q12	397	226	221	221
q13	17814	3039	3008	3008
q14	271	243	227	227
q15	512	481	483	481
q16	488	390	384	384
q17	993	743	687	687
q18	7418	6980	6823	6823
q19	1391	1011	1012	1011
q20	694	333	327	327
q21	4109	3124	3007	3007
q22	1129	1022	1026	1022
Total cold run time: 110688 ms
Total hot run time: 37994 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4357	4333	4268	4268
q2	385	267	277	267
q3	2915	2697	2671	2671
q4	1942	1652	1608	1608
q5	5659	5703	5744	5703
q6	232	142	134	134
q7	2222	1833	1824	1824
q8	3273	3450	3529	3450
q9	8910	8920	8828	8828
q10	3657	3437	3329	3329
q11	586	489	503	489
q12	844	675	665	665
q13	15317	3177	3282	3177
q14	327	286	287	286
q15	516	490	496	490
q16	507	453	452	452
q17	1844	1549	1520	1520
q18	8163	7756	7907	7756
q19	1747	1549	1518	1518
q20	2165	1914	1937	1914
q21	5592	5431	5470	5431
q22	1132	1062	1058	1058
Total cold run time: 72292 ms
Total hot run time: 56838 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 192471 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 4821ff3de3f7a150ad07611d03cb512254251e7b, data reload: false

query1	1266	887	875	875
query2	6349	1941	1921	1921
query3	10750	4020	4146	4020
query4	59698	27924	23019	23019
query5	5331	494	497	494
query6	402	165	161	161
query7	5768	287	292	287
query8	283	210	222	210
query9	8876	2524	2503	2503
query10	504	288	279	279
query11	17900	15113	15458	15113
query12	160	101	100	100
query13	1565	378	378	378
query14	10589	7262	7448	7262
query15	241	182	178	178
query16	7576	483	490	483
query17	1143	569	593	569
query18	2052	299	301	299
query19	295	160	149	149
query20	121	107	116	107
query21	208	102	100	100
query22	4812	4659	4528	4528
query23	34104	33724	33531	33531
query24	5948	2914	2879	2879
query25	555	409	396	396
query26	689	160	158	158
query27	1772	283	284	283
query28	3774	2074	2064	2064
query29	727	420	434	420
query30	241	155	153	153
query31	939	788	760	760
query32	85	57	61	57
query33	497	301	284	284
query34	864	468	474	468
query35	831	755	730	730
query36	1070	922	969	922
query37	142	86	87	86
query38	3992	3816	3866	3816
query39	1434	1407	1409	1407
query40	199	119	122	119
query41	49	48	47	47
query42	127	101	96	96
query43	521	472	486	472
query44	1099	753	759	753
query45	201	167	168	167
query46	1107	767	750	750
query47	1885	1798	1828	1798
query48	369	294	296	294
query49	774	441	442	441
query50	830	418	414	414
query51	7155	7036	6979	6979
query52	99	88	91	88
query53	255	186	179	179
query54	577	465	469	465
query55	78	81	80	80
query56	284	266	270	266
query57	1187	1079	1091	1079
query58	226	234	228	228
query59	2996	2967	2791	2791
query60	303	282	280	280
query61	125	123	123	123
query62	746	662	672	662
query63	225	184	184	184
query64	3939	2231	1883	1883
query65	3200	3208	3195	3195
query66	689	327	337	327
query67	15700	15276	15369	15276
query68	4980	546	561	546
query69	648	372	272	272
query70	1180	1099	1071	1071
query71	501	272	266	266
query72	6491	2309	2064	2064
query73	811	322	325	322
query74	9114	8881	8973	8881
query75	3468	2631	2720	2631
query76	3216	1080	955	955
query77	622	324	316	316
query78	10210	9083	9025	9025
query79	2078	538	554	538
query80	1532	497	488	488
query81	549	227	225	225
query82	418	136	130	130
query83	270	147	151	147
query84	261	79	75	75
query85	956	280	285	280
query86	377	304	292	292
query87	4391	4286	4244	4244
query88	3767	2351	2323	2323
query89	397	288	278	278
query90	1878	193	198	193
query91	124	98	97	97
query92	63	49	50	49
query93	2821	530	544	530
query94	703	291	306	291
query95	344	258	265	258
query96	612	272	271	271
query97	3178	3034	3090	3034
query98	216	222	208	208
query99	1589	1290	1286	1286
Total cold run time: 316263 ms
Total hot run time: 192471 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.06 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 4821ff3de3f7a150ad07611d03cb512254251e7b, data reload: false

query1	0.05	0.04	0.04
query2	0.08	0.04	0.04
query3	0.22	0.05	0.05
query4	1.68	0.07	0.08
query5	0.53	0.50	0.50
query6	1.15	0.73	0.73
query7	0.02	0.01	0.01
query8	0.05	0.04	0.05
query9	0.54	0.49	0.49
query10	0.55	0.54	0.54
query11	0.15	0.12	0.11
query12	0.15	0.12	0.13
query13	0.61	0.58	0.59
query14	0.76	0.79	0.80
query15	0.83	0.83	0.81
query16	0.37	0.37	0.38
query17	1.06	1.04	1.07
query18	0.21	0.21	0.22
query19	1.95	1.89	1.83
query20	0.01	0.01	0.01
query21	15.40	0.66	0.66
query22	3.59	7.69	1.96
query23	18.30	1.32	1.30
query24	2.12	0.24	0.21
query25	0.16	0.08	0.08
query26	0.28	0.18	0.19
query27	0.08	0.08	0.07
query28	13.22	1.01	1.01
query29	12.59	3.38	3.31
query30	0.24	0.05	0.05
query31	2.88	0.39	0.40
query32	3.26	0.48	0.47
query33	2.95	3.03	2.98
query34	17.02	4.46	4.49
query35	4.52	4.47	4.50
query36	0.66	0.48	0.47
query37	0.19	0.16	0.16
query38	0.16	0.15	0.15
query39	0.05	0.04	0.04
query40	0.17	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.04	0.04
query43	0.04	0.04	0.04
Total cold run time: 109 s
Total hot run time: 31.06 s

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 23, 2024

run cloud_p1

@luwei16
Copy link
Contributor Author

luwei16 commented Aug 23, 2024

run buildall

@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions GitHub Actions
Copy link
Contributor

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

TPC-H: Total hot run time: 38136 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b380de3b76021ba11a9aa569558b8983c445ceb1, data reload: false

------ Round 1 ----------------------------------
q1	18386	4498	4419	4419
q2	2699	198	177	177
q3	11221	1120	1068	1068
q4	10499	823	751	751
q5	7931	2854	2852	2852
q6	225	138	141	138
q7	980	622	632	622
q8	9343	2051	2060	2051
q9	7248	6551	6534	6534
q10	7007	2213	2244	2213
q11	491	241	251	241
q12	401	224	222	222
q13	19029	3123	3030	3030
q14	300	232	235	232
q15	519	479	484	479
q16	487	396	395	395
q17	989	659	662	659
q18	7365	6961	6802	6802
q19	1375	1016	1087	1016
q20	696	342	323	323
q21	3900	3054	2905	2905
q22	1125	1007	1032	1007
Total cold run time: 112216 ms
Total hot run time: 38136 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4352	4305	4303	4303
q2	389	285	279	279
q3	2918	2644	2683	2644
q4	2002	1733	1713	1713
q5	5404	5371	5424	5371
q6	219	128	130	128
q7	2141	1725	1758	1725
q8	3220	3339	3359	3339
q9	8511	8505	8447	8447
q10	3468	3191	3180	3180
q11	614	502	505	502
q12	775	604	611	604
q13	10766	3037	3060	3037
q14	303	273	267	267
q15	531	494	475	475
q16	474	427	422	422
q17	1796	1485	1489	1485
q18	7729	7408	7378	7378
q19	1678	1561	1546	1546
q20	2051	1809	1827	1809
q21	5403	5357	5233	5233
q22	1104	1051	1018	1018
Total cold run time: 65848 ms
Total hot run time: 54905 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 187902 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b380de3b76021ba11a9aa569558b8983c445ceb1, data reload: false

query1	920	380	358	358
query2	6473	1997	1970	1970
query3	6645	219	218	218
query4	34246	23220	23102	23102
query5	4163	513	487	487
query6	259	162	167	162
query7	4591	303	287	287
query8	242	206	210	206
query9	8531	2464	2459	2459
query10	429	263	261	261
query11	17131	15264	15048	15048
query12	151	97	96	96
query13	1616	385	367	367
query14	9647	7299	7121	7121
query15	270	166	178	166
query16	8048	458	475	458
query17	1568	569	561	561
query18	2127	297	297	297
query19	222	151	150	150
query20	115	116	110	110
query21	211	108	102	102
query22	4322	4152	4144	4144
query23	34210	33319	34016	33319
query24	11065	2924	2845	2845
query25	643	403	399	399
query26	1368	160	161	160
query27	2825	285	281	281
query28	7285	2057	2059	2057
query29	883	435	424	424
query30	306	160	156	156
query31	970	783	777	777
query32	104	61	60	60
query33	744	306	303	303
query34	901	482	479	479
query35	868	753	744	744
query36	1099	959	935	935
query37	153	86	86	86
query38	3987	3914	3940	3914
query39	1468	1409	1423	1409
query40	277	119	119	119
query41	48	48	45	45
query42	120	100	102	100
query43	515	471	483	471
query44	1185	756	773	756
query45	195	163	166	163
query46	1110	763	752	752
query47	1903	1776	1839	1776
query48	390	290	295	290
query49	1075	444	439	439
query50	826	413	417	413
query51	7279	7053	7067	7053
query52	104	92	90	90
query53	251	184	182	182
query54	838	471	457	457
query55	81	75	80	75
query56	295	257	271	257
query57	1202	1108	1095	1095
query58	235	232	246	232
query59	3048	2989	2905	2905
query60	313	293	287	287
query61	123	116	121	116
query62	800	648	690	648
query63	217	188	188	188
query64	6381	2447	1943	1943
query65	3365	3151	3174	3151
query66	1281	330	355	330
query67	15652	15248	15322	15248
query68	4731	569	555	555
query69	433	275	283	275
query70	1183	1131	1077	1077
query71	388	278	281	278
query72	7435	2329	2042	2042
query73	774	317	321	317
query74	9300	8842	8819	8819
query75	3677	2693	2715	2693
query76	3237	1017	963	963
query77	513	326	320	320
query78	9703	9481	9127	9127
query79	1721	529	522	522
query80	733	503	508	503
query81	520	225	223	223
query82	702	137	141	137
query83	179	152	151	151
query84	237	75	73	73
query85	1451	287	302	287
query86	432	304	301	301
query87	4380	4326	4354	4326
query88	3830	2443	2319	2319
query89	388	294	289	289
query90	1807	197	202	197
query91	128	104	100	100
query92	65	53	53	53
query93	1420	540	525	525
query94	729	294	293	293
query95	359	264	265	264
query96	590	266	271	266
query97	3187	3084	3067	3067
query98	227	199	203	199
query99	1490	1302	1282	1282
Total cold run time: 295864 ms
Total hot run time: 187902 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.66 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b380de3b76021ba11a9aa569558b8983c445ceb1, data reload: false

query1	0.05	0.05	0.04
query2	0.08	0.05	0.04
query3	0.23	0.05	0.05
query4	1.69	0.07	0.07
query5	0.50	0.49	0.50
query6	1.13	0.75	0.74
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.55	0.50	0.47
query10	0.55	0.53	0.53
query11	0.16	0.12	0.11
query12	0.15	0.12	0.12
query13	0.63	0.58	0.59
query14	0.77	0.80	0.78
query15	0.84	0.82	0.81
query16	0.37	0.36	0.36
query17	0.98	1.07	0.99
query18	0.21	0.20	0.20
query19	1.81	1.79	1.71
query20	0.02	0.01	0.01
query21	15.42	0.67	0.67
query22	4.65	6.79	1.84
query23	18.29	1.33	1.25
query24	2.08	0.23	0.23
query25	0.16	0.08	0.09
query26	0.27	0.18	0.18
query27	0.08	0.07	0.07
query28	13.28	1.00	0.99
query29	12.67	3.35	3.38
query30	0.25	0.06	0.05
query31	2.88	0.39	0.39
query32	3.27	0.50	0.47
query33	2.99	2.99	3.04
query34	16.94	4.42	4.43
query35	4.49	4.46	4.46
query36	0.66	0.47	0.48
query37	0.20	0.15	0.15
query38	0.15	0.14	0.14
query39	0.04	0.03	0.04
query40	0.16	0.13	0.13
query41	0.09	0.05	0.05
query42	0.06	0.05	0.05
query43	0.05	0.04	0.04
Total cold run time: 109.92 s
Total hot run time: 30.66 s

dataroaring
dataroaring approved these changes Aug 23, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@dataroaring dataroaring merged commit 7f5bfb0 into apache:master Aug 26, 2024
27 of 30 checks passed
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Aug 27, 2024
@luwei16
[Fix](tablet-meta) limit the data size of tablet meta (apache#39455)
89abbf7
@cambyzju cambyzju mentioned this pull request Aug 27, 2024
Merged
luwei16 added a commit to luwei16/incubator-doris that referenced this pull request Aug 27, 2024
@luwei16
[Fix](tablet-meta) limit the data size of tablet meta (apache#39455)
b0f4aa2
dataroaring pushed a commit that referenced this pull request Aug 27, 2024
@luwei16 @dataroaring
[Fix](tablet-meta) limit the data size of tablet meta (#39455)
d734e43
@luwei16 luwei16 mentioned this pull request Aug 27, 2024
Merged
dataroaring pushed a commit that referenced this pull request Aug 27, 2024
@luwei16
[Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39974)
cb312ca 
pick master #39455
dataroaring pushed a commit that referenced this pull request Sep 9, 2024
@luwei16
[Fix](tablet-meta) limit the data size of tablet meta (#39455) (#39967)
8503efd 
pick master #39455
@gavinchou gavinchou mentioned this pull request Oct 13, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dataroaring dataroaring dataroaring approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/2.0.15-merged dev/2.1.6-merged dev/3.0.2-merged reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
5 participants
@luwei16 @doris-robot @Yukang-Lian @dataroaring @wm1581066