Skip to content

[fix](deploy) K8s deploy manager cannot get group host info by endpoint#30055

Merged
adonis0147 merged 1 commit intoapache:branch-2.0from
zxealous:fix-k8s-deploy-2.0
Jan 19, 2024
Merged

[fix](deploy) K8s deploy manager cannot get group host info by endpoint#30055
adonis0147 merged 1 commit intoapache:branch-2.0from
zxealous:fix-k8s-deploy-2.0

Conversation

@zxealous
Copy link
Contributor

@zxealous zxealous commented Jan 17, 2024

cherry-pick #27813 from master

K8s deploy manager cannot get group host info by endpoint. If we get group host info by endpoint, there is no need to init statefulset.

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

…nt (apache#27813)

K8s deploy manager cannot get group host info by endpoint. If we get group host info by endpoint, there is no need to init statefulset.
@zxealous
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 49973 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fe850475b801bfb694ed5d4ac51f277ab976403f, data reload: false

------ Round 1 ----------------------------------
q1	17623	4372	4361	4361
q2	2069	145	140	140
q3	10566	1872	1917	1872
q4	10334	1272	1305	1272
q5	8329	3965	3964	3964
q6	230	126	122	122
q7	2019	1594	1623	1594
q8	9285	2688	2704	2688
q9	10755	10512	10584	10512
q10	8623	3477	3491	3477
q11	417	239	237	237
q12	463	293	308	293
q13	18349	3939	3975	3939
q14	348	321	335	321
q15	516	468	454	454
q16	697	592	594	592
q17	1138	962	949	949
q18	7133	6998	6941	6941
q19	1700	1546	1522	1522
q20	512	316	290	290
q21	4446	4068	4029	4029
q22	518	404	404	404
Total cold run time: 116070 ms
Total hot run time: 49973 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4330	4303	4261	4261
q2	316	220	226	220
q3	4167	4143	4141	4141
q4	2734	2733	2727	2727
q5	7257	7159	7165	7159
q6	231	119	116	116
q7	3273	2810	2846	2810
q8	4352	4428	4423	4423
q9	17077	16822	16773	16773
q10	4178	4243	4262	4243
q11	767	694	667	667
q12	1004	844	831	831
q13	6247	3694	3739	3694
q14	447	419	417	417
q15	481	457	462	457
q16	742	696	707	696
q17	4005	3805	3780	3780
q18	8735	8853	8839	8839
q19	1704	1684	1640	1640
q20	2377	2117	2078	2078
q21	8508	8401	8425	8401
q22	1005	972	910	910
Total cold run time: 83937 ms
Total hot run time: 79283 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 237876 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fe850475b801bfb694ed5d4ac51f277ab976403f, data reload: false

query1	932	394	383	383
query2	6553	2281	2053	2053
query3	6922	196	196	196
query4	20044	17983	17917	17917
query5	19728	6246	6201	6201
query6	278	216	221	216
query7	4153	287	294	287
query8	268	256	226	226
query9	3044	2666	2624	2624
query10	424	296	298	296
query11	11232	10653	10625	10625
query12	117	77	68	68
query13	5641	631	617	617
query14	17621	13023	13422	13023
query15	369	241	231	231
query16	6435	256	264	256
query17	3247	882	853	853
query18	2259	397	399	397
query19	211	140	146	140
query20	79	76	76	76
query21	189	92	89	89
query22	5283	5121	5082	5082
query23	32444	31965	31927	31927
query24	6906	6550	6455	6455
query25	524	442	430	430
query26	534	154	156	154
query27	1859	301	290	290
query28	6159	2223	2201	2201
query29	2866	2760	2609	2609
query30	249	162	158	158
query31	878	709	712	709
query32	64	57	58	57
query33	402	249	247	247
query34	857	466	469	466
query35	1064	868	917	868
query36	1748	1583	1465	1465
query37	87	61	62	61
query38	3091	2956	2914	2914
query39	1385	1317	1312	1312
query40	204	96	86	86
query41	33	30	30	30
query42	85	78	86	78
query43	684	617	651	617
query44	1140	706	719	706
query45	240	230	224	224
query46	1215	977	968	968
query47	1941	1910	1708	1708
query48	970	680	646	646
query49	613	362	367	362
query50	873	611	643	611
query51	5638	5551	5564	5551
query52	92	80	81	80
query53	442	324	315	315
query54	2410	2283	2252	2252
query55	87	78	82	78
query56	221	194	205	194
query57	1159	1090	1097	1090
query58	211	210	179	179
query59	3418	3330	3283	3283
query60	206	204	219	204
query61	84	82	86	82
query62	782	525	504	504
query63	467	338	331	331
query64	2002	1271	1269	1269
query65	3678	3577	3644	3577
query66	821	368	359	359
query67	15465	14839	14948	14839
query68	7577	640	644	640
query69	541	336	378	336
query70	1766	1645	1576	1576
query71	369	306	302	302
query72	4521	3446	3347	3347
query73	717	315	313	313
query74	6221	5804	5850	5804
query75	4259	3495	3462	3462
query76	4543	1189	1156	1156
query77	557	249	246	246
query78	31211	52396	48585	48585
query79	16158	902	661	661
query80	5193	397	381	381
query81	572	229	230	229
query82	1232	104	93	93
query83	413	140	129	129
query84	257	70	67	67
query85	2524	281	276	276
query86	453	400	403	400
query87	3265	2995	3031	2995
query88	6970	2262	2293	2262
query89	470	298	275	275
query90	2531	210	194	194
query91	156	131	140	131
query92	62	51	55	51
query93	6614	635	584	584
query94	1740	208	204	204
query95	1129	1079	1052	1052
query96	624	328	320	320
query97	6425	6355	6384	6355
query98	193	176	163	163
query99	3851	911	885	885
Total cold run time: 344441 ms
Total hot run time: 237876 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.73 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fe850475b801bfb694ed5d4ac51f277ab976403f, data reload: false

query1	0.02	0.02	0.01
query2	0.06	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.06	0.06
query5	0.54	0.53	0.52
query6	1.23	0.62	0.62
query7	0.02	0.02	0.01
query8	0.03	0.02	0.02
query9	0.52	0.48	0.48
query10	0.53	0.54	0.54
query11	0.11	0.08	0.09
query12	0.12	0.09	0.09
query13	0.62	0.61	0.61
query14	0.79	0.78	0.80
query15	0.78	0.76	0.75
query16	0.39	0.37	0.36
query17	1.01	1.01	1.01
query18	0.21	0.28	0.25
query19	1.94	1.83	1.89
query20	0.02	0.01	0.00
query21	15.48	0.57	0.57
query22	1.75	1.95	1.30
query23	17.36	0.99	0.84
query24	4.35	1.16	1.37
query25	2.06	0.16	0.14
query26	0.16	0.14	0.15
query27	0.15	0.14	0.14
query28	8.13	0.72	0.75
query29	12.65	2.35	2.31
query30	0.62	0.58	0.53
query31	2.82	0.38	0.38
query32	3.37	0.49	0.49
query33	3.05	3.07	3.05
query34	15.26	4.79	4.81
query35	4.83	4.84	4.83
query36	1.06	1.01	1.01
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.02	0.02
query43	0.03	0.02	0.01
Total cold run time: 104.49 s
Total hot run time: 30.73 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit fe850475b801bfb694ed5d4ac51f277ab976403f with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.8 seconds inserted 10000000 Rows, about 458K ops/s

Copy link
Contributor

@adonis0147 adonis0147 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2024
@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

@chenlinzhong chenlinzhong left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@adonis0147 adonis0147 merged commit d6a155b into apache:branch-2.0 Jan 19, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
…nt (apache#27813) (apache#30055)

cherry-pick apache#27813 from master

K8s deploy manager cannot get group host info by endpoint. If we get group host info by endpoint, there is no need to init statefulset.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants