[feature](Cloud) Add one config to control FE's startup with or without built in vault #33308

ByteYue · 2024-04-07T06:16:49Z

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

doris-robot · 2024-04-07T06:16:54Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

ByteYue · 2024-04-07T06:17:35Z

run buildall

github-actions · 2024-04-07T06:22:04Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-07T07:02:33Z

run buildall

github-actions · 2024-04-07T07:07:00Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-07T08:16:37Z

clang-tidy review says "All clean, LGTM! 👍"

cloud/src/meta-service/meta_service_resource.cpp

ByteYue · 2024-04-08T04:17:43Z

run buildall

github-actions · 2024-04-08T04:22:33Z

clang-tidy review says "All clean, LGTM! 👍"

doris-robot · 2024-04-08T04:39:20Z

TPC-H: Total hot run time: 39446 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit cf30609cfd24aeccb2a176b259040c3d975ceaff, data reload: false

------ Round 1 ----------------------------------
q1	17627	4457	4226	4226
q2	2007	189	186	186
q3	10605	1270	1263	1263
q4	11077	799	804	799
q5	7570	2789	2722	2722
q6	223	139	138	138
q7	974	546	550	546
q8	9253	2090	2052	2052
q9	8793	7278	7255	7255
q10	8650	3552	3554	3552
q11	464	245	230	230
q12	433	208	212	208
q13	18799	2987	2994	2987
q14	273	235	238	235
q15	514	475	482	475
q16	519	390	381	381
q17	983	569	647	569
q18	7354	6780	6765	6765
q19	1619	1505	1527	1505
q20	674	317	301	301
q21	3363	2750	2792	2750
q22	376	316	301	301
Total cold run time: 112150 ms
Total hot run time: 39446 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4359	4179	4332	4179
q2	373	275	264	264
q3	2994	2743	2713	2713
q4	1885	1608	1589	1589
q5	5322	5303	5290	5290
q6	208	122	126	122
q7	1790	1434	1435	1434
q8	3177	3383	3298	3298
q9	9329	9263	9334	9263
q10	3892	3716	3758	3716
q11	588	478	493	478
q12	754	597	612	597
q13	13808	3024	2941	2941
q14	294	260	265	260
q15	521	484	461	461
q16	468	426	430	426
q17	1780	1476	1476	1476
q18	7447	7424	7316	7316
q19	1631	1520	1527	1520
q20	1946	1753	1755	1753
q21	4846	4797	4876	4797
q22	541	466	463	463
Total cold run time: 67953 ms
Total hot run time: 54356 ms

doris-robot · 2024-04-08T04:51:12Z

TPC-DS: Total hot run time: 181954 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit cf30609cfd24aeccb2a176b259040c3d975ceaff, data reload: false

query1	907	1134	1118	1118
query2	7322	2032	1946	1946
query3	6649	205	213	205
query4	37827	21579	21313	21313
query5	4167	423	420	420
query6	222	180	179	179
query7	4037	291	287	287
query8	219	169	178	169
query9	5720	2307	2282	2282
query10	514	249	254	249
query11	14707	14160	14188	14160
query12	146	95	92	92
query13	998	360	364	360
query14	9110	6847	6892	6847
query15	218	179	179	179
query16	6690	268	256	256
query17	1492	578	546	546
query18	1360	299	271	271
query19	197	152	154	152
query20	102	87	88	87
query21	201	132	122	122
query22	4976	4821	4840	4821
query23	33756	33137	33024	33024
query24	12974	2892	2993	2892
query25	543	360	365	360
query26	1931	154	154	154
query27	3044	314	313	313
query28	7582	1890	1895	1890
query29	879	625	610	610
query30	309	172	166	166
query31	921	708	713	708
query32	61	56	59	56
query33	604	257	252	252
query34	883	486	487	486
query35	828	701	686	686
query36	1025	893	935	893
query37	233	74	81	74
query38	3513	3403	3452	3403
query39	1705	1525	1535	1525
query40	279	133	141	133
query41	50	47	50	47
query42	121	98	96	96
query43	482	468	469	468
query44	1381	716	716	716
query45	290	265	270	265
query46	1089	725	714	714
query47	1911	1839	1830	1830
query48	377	283	283	283
query49	1160	375	365	365
query50	753	381	396	381
query51	6692	6648	6562	6562
query52	109	95	102	95
query53	356	288	285	285
query54	277	238	246	238
query55	78	72	73	72
query56	255	229	238	229
query57	1197	1117	1127	1117
query58	238	215	210	210
query59	2699	2650	2692	2650
query60	260	252	267	252
query61	110	106	106	106
query62	652	450	443	443
query63	310	283	286	283
query64	5240	4017	4000	4000
query65	3120	3052	3034	3034
query66	1373	359	319	319
query67	15573	15178	14877	14877
query68	8466	543	543	543
query69	599	317	327	317
query70	1253	1154	1113	1113
query71	502	294	280	280
query72	6622	2686	2408	2408
query73	823	315	317	315
query74	7169	6506	6402	6402
query75	3586	2261	2318	2261
query76	4732	1071	1035	1035
query77	608	247	252	247
query78	10972	10389	10173	10173
query79	9276	524	508	508
query80	1948	434	422	422
query81	522	239	234	234
query82	1383	96	95	95
query83	230	164	167	164
query84	266	81	84	81
query85	1166	277	260	260
query86	464	327	310	310
query87	3772	3511	3555	3511
query88	6097	2276	2272	2272
query89	546	381	379	379
query90	1943	184	179	179
query91	121	93	92	92
query92	67	47	49	47
query93	7080	513	494	494
query94	1159	183	180	180
query95	406	312	303	303
query96	602	263	262	262
query97	2656	2438	2494	2438
query98	233	224	203	203
query99	1135	835	803	803
Total cold run time: 311408 ms
Total hot run time: 181954 ms

doris-robot · 2024-04-08T04:56:39Z

ClickBench: Total hot run time: 31.35 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit cf30609cfd24aeccb2a176b259040c3d975ceaff, data reload: false

query1	0.04	0.04	0.03
query2	0.08	0.03	0.04
query3	0.24	0.06	0.05
query4	1.68	0.08	0.08
query5	0.50	0.49	0.50
query6	1.12	0.66	0.65
query7	0.02	0.02	0.01
query8	0.04	0.04	0.04
query9	0.56	0.49	0.51
query10	0.53	0.57	0.54
query11	0.17	0.11	0.11
query12	0.15	0.12	0.12
query13	0.58	0.58	0.58
query14	0.78	0.78	0.78
query15	0.82	0.79	0.80
query16	0.35	0.36	0.35
query17	0.96	1.00	1.02
query18	0.21	0.24	0.26
query19	1.87	1.72	1.70
query20	0.02	0.01	0.01
query21	15.44	0.64	0.64
query22	3.62	5.82	3.02
query23	18.25	1.29	1.25
query24	1.91	0.23	0.21
query25	0.14	0.08	0.08
query26	0.27	0.16	0.16
query27	0.08	0.08	0.07
query28	13.40	0.98	0.98
query29	12.61	3.27	3.30
query30	0.26	0.07	0.06
query31	2.84	0.37	0.37
query32	3.32	0.46	0.45
query33	2.87	2.81	2.84
query34	17.04	4.41	4.41
query35	4.47	4.47	4.46
query36	0.66	0.45	0.46
query37	0.20	0.15	0.15
query38	0.14	0.13	0.13
query39	0.05	0.03	0.03
query40	0.16	0.14	0.14
query41	0.10	0.05	0.04
query42	0.06	0.05	0.05
query43	0.04	0.04	0.03
Total cold run time: 108.65 s
Total hot run time: 31.35 s

doris-robot · 2024-04-08T04:59:45Z

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit cf30609cfd24aeccb2a176b259040c3d975ceaff with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.4 seconds inserted 10000000 Rows, about 746K ops/s

ByteYue · 2024-04-08T08:30:53Z

run buildall

github-actions · 2024-04-08T08:34:39Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-08T10:18:00Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-08T10:25:01Z

run buildall

github-actions · 2024-04-08T10:26:25Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-08T11:17:47Z

run buildall

github-actions · 2024-04-08T11:22:54Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-08T11:23:23Z

clang-tidy review says "All clean, LGTM! 👍"

platoneko

FE 需要有个 Env 级属性 enableStorageVault，启动时获取，create tablets 需要用到来决定请求是否带上 vault name

cloud/src/meta-service/meta_service.cpp

cloud/src/meta-service/meta_service_resource.cpp

ByteYue · 2024-04-08T11:37:58Z

FE 需要有个 Env 级属性 enableStorageVault，启动时获取，create tablets 需要用到来决定请求是否带上 vault name

估计得在get cluster时获取local type里拿到这个属性

ByteYue · 2024-04-08T15:15:21Z

run buildall

github-actions · 2024-04-08T15:16:20Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-08T15:19:52Z

clang-tidy review says "All clean, LGTM! 👍"

ByteYue · 2024-04-09T02:21:23Z

run buildall

github-actions · 2024-04-09T02:26:01Z

clang-tidy review says "All clean, LGTM! 👍"

github-actions · 2024-04-09T04:54:47Z

PR approved by at least one committer and no changes requested.

github-actions · 2024-04-09T04:54:49Z

PR approved by anyone and no changes requested.

…ut built in vault (apache#33308)

ByteYue added 2 commits April 7, 2024 13:37

add one filed

54e1d55

add fe config

a50eb8d

github-actions bot added the meta-change label Apr 7, 2024

remove config

eec54b4

ByteYue force-pushed the dont_start_fe_when_no_built_in branch from 8c15880 to eec54b4 Compare April 7, 2024 08:11

w41ter reviewed Apr 8, 2024

View reviewed changes

cloud/src/meta-service/meta_service_resource.cpp Outdated Show resolved Hide resolved

fix comment

cf30609

add one bool in instancepb

c7c6a13

create tablets check

84994f3

tmp

279b663

ByteYue mentioned this pull request Apr 8, 2024

[feature](Cloud) Enhance for compatibility of old cloud create table logic #33318

Closed

update ut

1d13e2e

ByteYue force-pushed the dont_start_fe_when_no_built_in branch from b1a1352 to 1d13e2e Compare April 8, 2024 11:17

platoneko reviewed Apr 8, 2024

View reviewed changes

cloud/src/meta-service/meta_service.cpp Outdated Show resolved Hide resolved

cloud/src/meta-service/meta_service_resource.cpp Outdated Show resolved Hide resolved

ByteYue added 3 commits April 8, 2024 22:40

add it to resp

7476f91

add it in fe

461f29e

solve commet

fb7ba03

tmp

5406fda

platoneko approved these changes Apr 9, 2024

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Apr 9, 2024

github-actions bot added the reviewed label Apr 9, 2024

w41ter approved these changes Apr 9, 2024

View reviewed changes

dataroaring merged commit dc4d25f into apache:master Apr 9, 2024
25 of 29 checks passed

seawinde pushed a commit to seawinde/doris that referenced this pull request Apr 10, 2024

[feature](Cloud) Add one config to control FE's startup with or witho…

ea3e0b2

…ut built in vault (apache#33308)

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[feature](Cloud) Add one config to control FE's startup with or without built in vault #33308

[feature](Cloud) Add one config to control FE's startup with or without built in vault #33308

ByteYue commented Apr 7, 2024

doris-robot commented Apr 7, 2024

ByteYue commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

ByteYue commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

platoneko left a comment

ByteYue commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

[feature](Cloud) Add one config to control FE's startup with or without built in vault #33308

[feature](Cloud) Add one config to control FE's startup with or without built in vault #33308

Conversation

ByteYue commented Apr 7, 2024

Proposed changes

Further comments

doris-robot commented Apr 7, 2024

ByteYue commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

ByteYue commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

github-actions bot commented Apr 7, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

doris-robot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

platoneko left a comment

Choose a reason for hiding this comment

ByteYue commented Apr 8, 2024

ByteYue commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

github-actions bot commented Apr 8, 2024

ByteYue commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024

github-actions bot commented Apr 9, 2024