Skip to content

[fix](allocation) fix two problems for force_olap_table_replication_allocation #34554#34659

Merged
cambyzju merged 1 commit intoapache:branch-2.0from
cambyzju:fix-2.0-0508
May 13, 2024
Merged

[fix](allocation) fix two problems for force_olap_table_replication_allocation #34554#34659
cambyzju merged 1 commit intoapache:branch-2.0from
cambyzju:fix-2.0-0508

Conversation

@cambyzju
Copy link
Contributor

@cambyzju cambyzju commented May 10, 2024

cherry pick #34554 to branch-2.0

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented May 10, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@cambyzju
Copy link
Contributor Author

cambyzju commented May 10, 2024

run buildall

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-H: Total hot run time: 49551 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 582f1d5a8fc67953df3e25204568c822d6b8cbac, data reload: false

------ Round 1 ----------------------------------
q1	17655	4354	4303	4303
q2	2025	152	144	144
q3	10480	1913	1939	1913
q4	10309	1254	1324	1254
q5	8775	3870	3925	3870
q6	237	124	127	124
q7	2064	1563	1591	1563
q8	9279	2700	2704	2700
q9	10505	10371	10219	10219
q10	8619	3467	3494	3467
q11	419	241	249	241
q12	468	303	312	303
q13	18359	3956	4006	3956
q14	348	338	342	338
q15	504	466	447	447
q16	686	583	577	577
q17	1130	972	963	963
q18	7377	6917	6863	6863
q19	1694	1536	1517	1517
q20	530	311	308	308
q21	4436	4090	4097	4090
q22	491	391	399	391
Total cold run time: 116390 ms
Total hot run time: 49551 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4276	4307	4305	4305
q2	323	223	224	223
q3	4132	4179	4121	4121
q4	2770	2749	2740	2740
q5	7123	7079	7084	7079
q6	236	118	121	118
q7	3269	2793	2822	2793
q8	4333	4447	4466	4447
q9	16797	16755	16673	16673
q10	4188	4238	4264	4238
q11	754	680	682	680
q12	1020	843	852	843
q13	5342	3744	3739	3739
q14	454	415	411	411
q15	527	459	459	459
q16	749	683	674	674
q17	3837	3848	3803	3803
q18	8848	8823	8837	8823
q19	1699	1691	1640	1640
q20	2393	2122	2101	2101
q21	8420	8528	8383	8383
q22	1001	921	927	921
Total cold run time: 82491 ms
Total hot run time: 79214 ms

@doris-robot
Copy link

doris-robot commented May 10, 2024

TPC-DS: Total hot run time: 202456 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 582f1d5a8fc67953df3e25204568c822d6b8cbac, data reload: false

query1	941	387	379	379
query2	6536	2769	2456	2456
query3	6921	209	206	206
query4	20797	17944	17937	17937
query5	19726	6572	6485	6485
query6	270	223	229	223
query7	4268	291	303	291
query8	270	255	237	237
query9	3102	2698	2618	2618
query10	425	309	276	276
query11	11357	10692	10704	10692
query12	122	73	69	69
query13	5582	679	664	664
query14	17369	13191	13236	13191
query15	375	223	220	220
query16	6458	274	256	256
query17	1721	1450	874	874
query18	2316	411	393	393
query19	208	145	147	145
query20	79	76	77	76
query21	201	95	95	95
query22	5282	5138	5047	5047
query23	32652	31884	31880	31880
query24	6991	6489	6497	6489
query25	529	415	412	412
query26	523	164	157	157
query27	1920	291	288	288
query28	6204	2383	2354	2354
query29	2840	2852	2821	2821
query30	240	163	163	163
query31	911	754	771	754
query32	59	64	59	59
query33	393	252	260	252
query34	835	467	483	467
query35	1118	920	872	872
query36	1376	1169	1378	1169
query37	89	57	59	57
query38	3128	2896	2905	2896
query39	1371	1340	1313	1313
query40	205	98	97	97
query41	42	36	35	35
query42	89	79	86	79
query43	862	781	608	608
query44	1142	706	717	706
query45	241	234	224	224
query46	1226	982	971	971
query47	1848	1781	1732	1732
query48	1016	708	705	705
query49	624	377	376	376
query50	861	606	626	606
query51	4826	4668	4718	4668
query52	97	79	76	76
query53	450	316	323	316
query54	2650	2457	2455	2455
query55	83	88	73	73
query56	208	200	217	200
query57	1196	1133	1084	1084
query58	211	190	199	190
query59	4093	3977	3879	3879
query60	206	192	220	192
query61	87	81	84	81
query62	836	526	476	476
query63	479	336	332	332
query64	2519	1514	1356	1356
query65	3650	3565	3555	3555
query66	766	371	403	371
query67	15264	15195	15470	15195
query68	8392	649	648	648
query69	568	353	348	348
query70	1446	1319	1317	1317
query71	400	299	308	299
query72	6411	3388	3450	3388
query73	739	335	327	327
query74	6340	5947	5841	5841
query75	4903	3659	3719	3659
query76	5011	1161	1197	1161
query77	713	263	246	246
query78	12532	12743	11698	11698
query79	11830	651	633	633
query80	813	391	388	388
query81	509	231	232	231
query82	1190	96	93	93
query83	165	133	134	133
query84	261	69	66	66
query85	842	298	296	296
query86	344	295	314	295
query87	3255	2999	3002	2999
query88	5171	2364	2363	2363
query89	431	274	275	274
query90	1961	207	219	207
query91	165	153	134	134
query92	57	52	53	52
query93	5787	577	563	563
query94	725	201	204	201
query95	1097	1039	1047	1039
query96	661	327	335	327
query97	6507	6406	6322	6322
query98	205	174	171	171
query99	2865	906	866	866
Total cold run time: 313578 ms
Total hot run time: 202456 ms

@doris-robot
Copy link

doris-robot commented May 10, 2024

ClickBench: Total hot run time: 30.32 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 582f1d5a8fc67953df3e25204568c822d6b8cbac, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.06
query4	1.76	0.07	0.07
query5	0.53	0.52	0.52
query6	1.22	0.61	0.62
query7	0.01	0.00	0.01
query8	0.04	0.02	0.03
query9	0.52	0.47	0.48
query10	0.54	0.54	0.52
query11	0.11	0.08	0.08
query12	0.11	0.09	0.09
query13	0.61	0.62	0.60
query14	0.78	0.80	0.79
query15	0.77	0.76	0.75
query16	0.36	0.40	0.37
query17	0.99	1.01	1.01
query18	0.21	0.26	0.25
query19	1.87	1.88	1.85
query20	0.01	0.00	0.00
query21	15.50	0.66	0.56
query22	2.00	2.40	1.49
query23	17.33	1.04	0.97
query24	6.09	0.61	1.25
query25	0.39	0.10	0.05
query26	0.64	0.15	0.16
query27	0.04	0.03	0.04
query28	7.23	0.72	0.75
query29	12.61	2.34	2.36
query30	0.60	0.52	0.51
query31	2.82	0.38	0.38
query32	3.39	0.49	0.49
query33	3.00	3.06	3.07
query34	15.28	4.79	4.79
query35	4.86	4.87	4.82
query36	1.06	1.00	1.03
query37	0.06	0.04	0.05
query38	0.03	0.02	0.02
query39	0.01	0.01	0.02
query40	0.16	0.14	0.14
query41	0.07	0.02	0.02
query42	0.02	0.02	0.01
query43	0.03	0.02	0.02
Total cold run time: 104 s
Total hot run time: 30.32 s

@doris-robot
Copy link

doris-robot commented May 10, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 582f1d5a8fc67953df3e25204568c822d6b8cbac with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.4 seconds inserted 10000000 Rows, about 467K ops/s

lide-reed
lide-reed approved these changes May 11, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions
Copy link
Contributor

github-actions bot commented May 11, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 11, 2024
@github-actions
Copy link
Contributor

github-actions bot commented May 11, 2024

PR approved by anyone and no changes requested.

xy720
xy720 approved these changes May 11, 2024
View reviewed changes
Copy link
Member

@xy720 xy720 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@cambyzju cambyzju merged commit a3d1854 into apache:branch-2.0 May 13, 2024
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@cambyzju
[fix](replication_allocation) fix two problems for force_olap_table_r…
bd047a7 
…eplication_allocation apache#34554 (apache#34659)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@xy720 xy720 xy720 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cambyzju @doris-robot @lide-reed @xy720