Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[enhancement](backup-restore) add config for upload/download task num per be (#27772) #34019

Merged
merged 1 commit into from
Apr 24, 2024
Merged

[enhancement](backup-restore) add config for upload/download task num per be (#27772) #34019

merged 1 commit into from
Apr 24, 2024

Conversation

nextdreamblue
Copy link
Contributor

@nextdreamblue nextdreamblue commented Apr 23, 2024 •
edited
Loading

set upload/download task num per be, and improve the overall speed of upload/download, enhance the performance of backup and recovery.

Proposed changes

Issue Number: close #27772

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@nextdreamblue
[enhancement](backup-restore) add config for upload/download task num…
60c86be 
… per be (apache#27772)

set upload/download task num per be, and improve the overall speed of upload/download, enhance the performance of backup and recovery.

---------

Signed-off-by: nextdreamblue <zxw520blue1@163.com>
@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@github-actions github-actions bot added the kind/docs Categorizes issue or PR as related to documentation. label Apr 23, 2024
@nextdreamblue
Copy link
Contributor Author

run buildall

@nextdreamblue nextdreamblue marked this pull request as ready for review April 23, 2024 08:39
lide-reed
lide-reed approved these changes Apr 23, 2024
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Apr 23, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@doris-robot
Copy link

TPC-H: Total hot run time: 49913 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 60c86befb8860b3f2187e58588770c38bc42cf5d, data reload: false

------ Round 1 ----------------------------------
q1	17765	4410	4317	4317
q2	2036	150	147	147
q3	10276	1929	1928	1928
q4	10124	1250	1357	1250
q5	8426	3905	4005	3905
q6	229	120	121	120
q7	2043	1585	1575	1575
q8	9514	2735	2723	2723
q9	14086	10466	10448	10448
q10	8603	3536	3528	3528
q11	406	233	242	233
q12	465	301	298	298
q13	18344	3955	3996	3955
q14	357	317	331	317
q15	495	453	456	453
q16	703	603	589	589
q17	1129	930	894	894
q18	7218	6877	6944	6877
q19	1687	1603	1565	1565
q20	541	311	292	292
q21	4460	4100	4087	4087
q22	516	412	415	412
Total cold run time: 119423 ms
Total hot run time: 49913 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4318	4336	4340	4336
q2	322	228	226	226
q3	4169	4144	4120	4120
q4	2751	2745	2763	2745
q5	7271	7213	7205	7205
q6	232	119	118	118
q7	3227	2838	2807	2807
q8	4369	4460	4465	4460
q9	16966	16880	17060	16880
q10	4200	4277	4238	4238
q11	732	681	666	666
q12	1021	873	870	870
q13	6804	3736	3769	3736
q14	461	430	431	430
q15	503	450	456	450
q16	751	683	719	683
q17	3804	3920	3857	3857
q18	8718	8899	8816	8816
q19	1720	1677	1680	1677
q20	2348	2143	2073	2073
q21	8562	8463	8559	8463
q22	1044	927	919	919
Total cold run time: 84293 ms
Total hot run time: 79775 ms

cambyzju
cambyzju approved these changes Apr 23, 2024
View reviewed changes
Copy link
Contributor

@cambyzju cambyzju left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@doris-robot
Copy link

TPC-DS: Total hot run time: 202342 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 60c86befb8860b3f2187e58588770c38bc42cf5d, data reload: false

query1	906	386	374	374
query2	6542	2858	2522	2522
query3	6922	199	193	193
query4	21246	17979	17887	17887
query5	19717	6472	6513	6472
query6	297	211	224	211
query7	4159	295	301	295
query8	247	268	231	231
query9	3161	2729	2658	2658
query10	415	296	301	296
query11	11321	10686	10757	10686
query12	120	76	70	70
query13	5587	696	702	696
query14	17806	13501	13584	13501
query15	365	239	232	232
query16	6487	269	263	263
query17	1749	1451	860	860
query18	2321	405	406	405
query19	209	144	138	138
query20	70	75	73	73
query21	196	96	92	92
query22	5310	4964	5044	4964
query23	32604	31815	31883	31815
query24	6940	6540	6474	6474
query25	512	415	412	412
query26	524	159	161	159
query27	1872	295	289	289
query28	6156	2396	2354	2354
query29	2889	2808	2875	2808
query30	243	159	163	159
query31	888	734	767	734
query32	64	58	57	57
query33	401	251	253	251
query34	845	456	471	456
query35	1113	945	945	945
query36	1271	1013	1236	1013
query37	92	63	61	61
query38	3122	2914	2909	2909
query39	1353	1317	1318	1317
query40	208	91	93	91
query41	49	34	35	34
query42	90	87	80	80
query43	732	740	647	647
query44	1140	738	742	738
query45	234	224	224	224
query46	1248	967	976	967
query47	1916	1730	1806	1730
query48	1015	721	702	702
query49	621	367	369	367
query50	866	636	591	591
query51	4716	4619	4688	4619
query52	85	74	69	69
query53	439	315	311	311
query54	2632	2480	2458	2458
query55	83	76	85	76
query56	216	197	207	197
query57	1249	1193	1159	1159
query58	222	190	190	190
query59	3978	3898	4284	3898
query60	206	192	197	192
query61	87	80	82	80
query62	879	483	506	483
query63	469	332	346	332
query64	2444	1501	1505	1501
query65	3635	3554	3580	3554
query66	813	373	379	373
query67	15581	15896	15001	15001
query68	9113	670	687	670
query69	584	344	352	344
query70	1903	1424	1587	1424
query71	402	299	299	299
query72	6558	3452	3406	3406
query73	728	334	326	326
query74	6307	5836	5815	5815
query75	5315	3691	3568	3568
query76	5796	1174	1210	1174
query77	930	258	249	249
query78	12700	11946	11407	11407
query79	10186	655	647	647
query80	1557	389	377	377
query81	490	230	227	227
query82	1604	105	95	95
query83	168	128	138	128
query84	254	68	67	67
query85	849	291	283	283
query86	347	298	297	297
query87	3252	3039	2995	2995
query88	5026	2325	2327	2325
query89	484	280	284	280
query90	1963	208	214	208
query91	182	146	132	132
query92	55	49	54	49
query93	7089	550	593	550
query94	701	197	202	197
query95	1108	1051	1062	1051
query96	651	321	319	319
query97	6499	6352	6371	6352
query98	182	167	167	167
query99	3108	918	1014	918
Total cold run time: 317986 ms
Total hot run time: 202342 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.48 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 60c86befb8860b3f2187e58588770c38bc42cf5d, data reload: false

query1	0.03	0.02	0.02
query2	0.07	0.02	0.02
query3	0.25	0.04	0.05
query4	1.80	0.08	0.06
query5	0.54	0.53	0.52
query6	1.28	0.61	0.61
query7	0.02	0.02	0.01
query8	0.04	0.02	0.02
query9	0.52	0.48	0.47
query10	0.55	0.53	0.53
query11	0.13	0.08	0.09
query12	0.11	0.08	0.08
query13	0.61	0.61	0.62
query14	0.78	0.79	0.79
query15	0.78	0.75	0.75
query16	0.36	0.39	0.36
query17	0.97	1.01	1.01
query18	0.25	0.25	0.24
query19	1.94	1.87	1.79
query20	0.02	0.01	0.01
query21	15.46	0.57	0.55
query22	2.11	2.47	1.38
query23	16.79	0.95	0.99
query24	4.35	1.04	1.15
query25	0.32	0.11	0.06
query26	0.58	0.14	0.15
query27	0.05	0.04	0.03
query28	8.60	0.75	0.71
query29	12.61	2.35	2.31
query30	0.56	0.51	0.51
query31	2.80	0.39	0.37
query32	3.38	0.50	0.49
query33	3.04	3.11	3.07
query34	15.27	4.78	4.77
query35	4.86	4.84	4.83
query36	1.10	1.01	1.04
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.02	0.01
query40	0.16	0.14	0.14
query41	0.07	0.01	0.01
query42	0.02	0.01	0.01
query43	0.02	0.01	0.02
Total cold run time: 103.32 s
Total hot run time: 30.48 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 60c86befb8860b3f2187e58588770c38bc42cf5d with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

@lide-reed lide-reed merged commit a71f75a into apache:branch-2.0 Apr 24, 2024
28 of 31 checks passed
mongo360 pushed a commit to mongo360/doris that referenced this pull request Aug 16, 2024
@nextdreamblue
[enhancement](backup-restore) add config for upload/download task num…
7abcbe7 
… per be (apache#27772) (apache#34019)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@cambyzju cambyzju cambyzju approved these changes

@lide-reed lide-reed lide-reed approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. kind/docs Categorizes issue or PR as related to documentation. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@nextdreamblue @doris-robot @lide-reed @cambyzju