Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[improve] Make the schema change memory space adaptive #34515

Merged
merged 1 commit into from
May 8, 2024
Merged

[improve] Make the schema change memory space adaptive #34515

merged 1 commit into from
May 8, 2024

Conversation

Lchangliang
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

github-actions[bot]
github-actions bot reviewed May 8, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/olap/storage_engine.h
@@ -233,6 +233,8 @@ class StorageEngine {

std::set<string> get_broken_paths() { return _broken_paths; }

int64_t memory_limitation_bytes_per_thread_for_schema_change() const;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function 'memory_limitation_bytes_per_thread_for_schema_change' should be marked [[nodiscard]] [modernize-use-nodiscard]

Suggested change
int64_t memory_limitation_bytes_per_thread_for_schema_change() const;
[[nodiscard]] int64_t memory_limitation_bytes_per_thread_for_schema_change() const;

@Lchangliang
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 50182 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit f739f98b68a7ba7271c8e038fbbdf99612b8ce37, data reload: false

------ Round 1 ----------------------------------
q1	17539	4359	4334	4334
q2	2030	162	142	142
q3	10265	1960	1952	1952
q4	10105	1241	1310	1241
q5	8452	4153	3943	3943
q6	238	123	123	123
q7	2049	1601	1566	1566
q8	9301	2752	2818	2752
q9	11092	11129	10571	10571
q10	8703	3536	3541	3536
q11	421	235	237	235
q12	463	302	311	302
q13	18341	3975	4093	3975
q14	355	321	324	321
q15	503	461	458	458
q16	688	568	569	568
q17	1138	968	981	968
q18	7395	6874	7040	6874
q19	1705	1561	1514	1514
q20	539	326	325	325
q21	4511	4160	4091	4091
q22	497	393	391	391
Total cold run time: 116330 ms
Total hot run time: 50182 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4300	4367	4338	4338
q2	319	216	222	216
q3	4177	4153	4128	4128
q4	2775	2753	2760	2753
q5	7180	7147	7177	7147
q6	233	120	118	118
q7	3290	2833	2808	2808
q8	4352	4521	4481	4481
q9	16909	16835	16738	16738
q10	4236	4311	4233	4233
q11	763	670	707	670
q12	1019	845	851	845
q13	4910	3737	3753	3737
q14	453	420	414	414
q15	504	461	446	446
q16	738	682	672	672
q17	3916	3854	3879	3854
q18	8776	8746	8727	8727
q19	1731	1730	1670	1670
q20	2391	2177	2113	2113
q21	8491	8574	8507	8507
q22	1049	932	989	932
Total cold run time: 82512 ms
Total hot run time: 79547 ms

@doris-robot
Copy link

TeamCity be ut coverage result:
Function Coverage: 37.81% (8078/21365)
Line Coverage: 29.45% (65939/223899)
Region Coverage: 28.92% (33946/117383)
Branch Coverage: 24.77% (17418/70308)
Coverage Report: http://coverage.selectdb-in.cc/coverage/f739f98b68a7ba7271c8e038fbbdf99612b8ce37_f739f98b68a7ba7271c8e038fbbdf99612b8ce37/report/index.html

@doris-robot
Copy link

TPC-DS: Total hot run time: 202312 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit f739f98b68a7ba7271c8e038fbbdf99612b8ce37, data reload: false

query1	922	391	374	374
query2	6524	2919	2729	2729
query3	6910	208	203	203
query4	19900	17986	17823	17823
query5	19731	6507	6438	6438
query6	281	218	222	218
query7	4155	305	299	299
query8	265	232	221	221
query9	3128	2710	2620	2620
query10	405	294	318	294
query11	11352	10705	10746	10705
query12	114	80	72	72
query13	5572	691	651	651
query14	17818	13949	13538	13538
query15	356	227	229	227
query16	6453	275	261	261
query17	1728	1456	870	870
query18	2313	401	397	397
query19	201	142	147	142
query20	79	70	76	70
query21	190	96	87	87
query22	5239	4854	5025	4854
query23	32638	31917	31701	31701
query24	7040	6509	6519	6509
query25	512	433	435	433
query26	632	164	155	155
query27	2057	293	291	291
query28	6201	2356	2321	2321
query29	2867	2752	2823	2752
query30	233	160	159	159
query31	890	757	732	732
query32	65	59	64	59
query33	395	250	250	250
query34	835	468	467	467
query35	1118	907	818	818
query36	1303	1287	1146	1146
query37	86	66	59	59
query38	3058	2898	2926	2898
query39	1358	1322	1319	1319
query40	272	92	94	92
query41	37	36	34	34
query42	90	84	83	83
query43	759	735	753	735
query44	1112	710	709	709
query45	246	230	229	229
query46	1215	969	971	969
query47	2100	1716	1924	1716
query48	1005	701	685	685
query49	631	378	363	363
query50	866	602	600	600
query51	4702	4666	4644	4644
query52	91	76	81	76
query53	439	316	310	310
query54	2667	2428	2475	2428
query55	91	84	72	72
query56	192	209	202	202
query57	1341	1198	1105	1105
query58	208	194	201	194
query59	4071	3973	4018	3973
query60	209	200	202	200
query61	84	82	82	82
query62	776	468	493	468
query63	482	336	333	333
query64	2545	1469	1430	1430
query65	3623	3550	3540	3540
query66	803	375	376	375
query67	15996	15135	15428	15135
query68	9546	638	663	638
query69	580	352	339	339
query70	1753	1419	1366	1366
query71	424	306	302	302
query72	6470	3394	3439	3394
query73	745	318	319	318
query74	6419	5884	5837	5837
query75	5329	3690	3585	3585
query76	5798	1149	1199	1149
query77	1011	249	253	249
query78	12652	11583	12037	11583
query79	7636	642	656	642
query80	1003	385	398	385
query81	455	231	227	227
query82	806	102	96	96
query83	165	132	130	130
query84	251	69	70	69
query85	848	293	290	290
query86	336	301	297	297
query87	3216	3039	3035	3035
query88	4477	2351	2384	2351
query89	391	290	330	290
query90	1896	203	203	203
query91	167	140	136	136
query92	60	53	50	50
query93	6369	588	560	560
query94	695	200	196	196
query95	1097	1039	1052	1039
query96	652	339	329	329
query97	6509	6323	6356	6323
query98	187	177	170	170
query99	2847	790	889	790
Total cold run time: 312367 ms
Total hot run time: 202312 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 31.27 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit f739f98b68a7ba7271c8e038fbbdf99612b8ce37, data reload: false

query1	0.02	0.03	0.02
query2	0.06	0.03	0.02
query3	0.25	0.05	0.06
query4	1.78	0.07	0.07
query5	0.53	0.52	0.52
query6	1.23	0.62	0.60
query7	0.02	0.01	0.01
query8	0.03	0.02	0.03
query9	0.53	0.46	0.49
query10	0.54	0.54	0.53
query11	0.12	0.09	0.09
query12	0.10	0.09	0.08
query13	0.62	0.61	0.62
query14	0.79	0.78	0.79
query15	0.80	0.76	0.78
query16	0.39	0.36	0.35
query17	0.98	0.99	1.00
query18	0.23	0.24	0.25
query19	1.93	1.87	1.86
query20	0.01	0.01	0.01
query21	15.49	0.56	0.55
query22	1.70	2.71	1.95
query23	17.22	1.03	0.89
query24	5.26	1.42	1.18
query25	0.39	0.08	0.05
query26	0.58	0.16	0.15
query27	0.04	0.05	0.04
query28	7.64	0.72	0.72
query29	12.69	2.33	2.32
query30	0.62	0.52	0.53
query31	2.82	0.39	0.37
query32	3.37	0.50	0.50
query33	3.07	3.11	3.05
query34	15.26	4.79	4.80
query35	4.87	4.88	4.84
query36	1.04	1.02	1.03
query37	0.06	0.04	0.04
query38	0.03	0.02	0.02
query39	0.02	0.01	0.01
query40	0.17	0.14	0.14
query41	0.06	0.02	0.02
query42	0.02	0.02	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.4 s
Total hot run time: 31.27 s

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit f739f98b68a7ba7271c8e038fbbdf99612b8ce37 with default session variables
Stream load json:         20 seconds loaded 2358488459 Bytes, about 112 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       21.5 seconds inserted 10000000 Rows, about 465K ops/s

dataroaring
dataroaring approved these changes May 8, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label May 8, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 8, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented May 8, 2024

PR approved by anyone and no changes requested.

@xiaokang xiaokang merged commit 778a887 into apache:branch-2.0 May 8, 2024
24 of 28 checks passed
@xiaokang
Copy link
Contributor

xiaokang commented May 8, 2024

#34350

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions left review comments

@dataroaring dataroaring dataroaring approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@Lchangliang @doris-robot @xiaokang @dataroaring