Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[revert](cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" #38828

Merged
merged 2 commits into from
Aug 4, 2024
Merged

[revert](cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" #38828

merged 2 commits into from
Aug 4, 2024

Conversation

gavinchou
Copy link
Contributor

@gavinchou gavinchou commented Aug 4, 2024 •
edited
Loading

We have to figure out why it causes a SEGV when running cloud_p0 later

#0 doris::cloud::TabletCompactionJobPB::_internal_input_versions(int) const
        /root/doris/cloud/../gensrc/build/gen_cpp/cloud.pb.h:48193:33
#1 doris::cloud::MetaServiceImpl::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*)
        /root/doris/cloud/src/meta-service/meta_service_job.cpp:436:9
#2 void doris::cloud::MetaServiceProxy::call_impl<doris::cloud::StartTabletJobRequest, doris::cloud::StartTabletJobResponse>(void (doris::cloud::MetaService::*)(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*), google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*)
        /root/doris/cloud/src/meta-service/meta_service.h:684:13
#3 doris::cloud::MetaServiceProxy::start_tablet_job(google::protobuf::RpcController*, doris::cloud::StartTabletJobRequest const*, doris::cloud::StartTabletJobResponse*, google::protobuf::Closure*)
        /root/doris/cloud/src/meta-service/meta_service.h:478:9
#4 doris::cloud::MetaService::CallMethod(google::protobuf::MethodDescriptor const*, google::protobuf::RpcController*, google::protobuf::Message const*, google::protobuf::Message*, google::protobuf::Closure*)
        /root/doris/gensrc/build/gen_cpp/cloud.pb.cc:0:7

This PR also add some FE log

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@gavinchou
Copy link
Contributor Author

run cloud_p0

github-actions[bot]
github-actions bot reviewed Aug 4, 2024
View reviewed changes
Copy link
Contributor

@github-actions github-actions bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

clang-tidy made some suggestions

be/src/cloud/cloud_schema_change_job.cpp
}

Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params,
cloud::TabletJobInfoPB& job) {
Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

warning: function '_convert_historical_rowsets' exceeds recommended size/complexity thresholds [readability-function-size]

Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params) {
                             ^
Additional context

be/src/cloud/cloud_schema_change_job.cpp:181: 205 lines including whitespace and comments (threshold 80)

Status CloudSchemaChangeJob::_convert_historical_rowsets(const SchemaChangeParams& sc_params) {
                             ^

@gavinchou
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 41729 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false

------ Round 1 ----------------------------------
q1	17631	4383	4101	4101
q2	2028	202	196	196
q3	10607	1301	1317	1301
q4	10227	812	908	812
q5	7651	3023	2993	2993
q6	224	142	139	139
q7	1045	605	607	605
q8	9433	1958	1986	1958
q9	8453	6616	6602	6602
q10	8709	3864	3879	3864
q11	428	255	248	248
q12	420	225	222	222
q13	17773	2919	2949	2919
q14	273	240	251	240
q15	515	500	493	493
q16	546	400	392	392
q17	987	941	950	941
q18	8000	7307	7221	7221
q19	1485	1237	1224	1224
q20	558	323	343	323
q21	5326	4655	4780	4655
q22	357	283	280	280
Total cold run time: 112676 ms
Total hot run time: 41729 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4113	4001	3988	3988
q2	326	227	217	217
q3	2979	3001	3149	3001
q4	2005	2040	1961	1961
q5	5627	5501	5448	5448
q6	222	135	133	133
q7	2070	1733	1827	1733
q8	3327	3399	3373	3373
q9	8690	8671	8827	8671
q10	3962	4074	3917	3917
q11	565	443	456	443
q12	747	624	598	598
q13	15312	3135	3124	3124
q14	313	279	280	279
q15	529	484	485	484
q16	459	411	424	411
q17	1803	1756	1773	1756
q18	8250	7802	7854	7802
q19	1877	1739	1740	1739
q20	2115	1850	1869	1850
q21	5811	5388	5381	5381
q22	537	458	471	458
Total cold run time: 71639 ms
Total hot run time: 56767 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 169641 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false

query1	924	376	369	369
query2	6458	1647	1667	1647
query3	6671	210	227	210
query4	20122	17423	17127	17127
query5	3634	514	504	504
query6	268	193	174	174
query7	4607	300	291	291
query8	253	194	191	191
query9	8522	2349	2338	2338
query10	446	296	290	290
query11	10394	10106	10049	10049
query12	122	89	89	89
query13	1628	383	381	381
query14	9511	7242	7777	7242
query15	217	162	167	162
query16	6937	439	401	401
query17	971	592	584	584
query18	1495	302	285	285
query19	193	146	148	146
query20	99	93	89	89
query21	205	99	98	98
query22	4217	4276	3947	3947
query23	33859	33603	33691	33603
query24	10239	3096	3086	3086
query25	684	404	405	404
query26	1627	164	158	158
query27	2912	285	293	285
query28	7193	1996	1994	1994
query29	1065	447	445	445
query30	240	156	156	156
query31	950	762	758	758
query32	105	55	56	55
query33	671	314	319	314
query34	940	507	518	507
query35	906	750	764	750
query36	1052	897	882	882
query37	224	85	83	83
query38	2974	2878	2805	2805
query39	875	819	798	798
query40	290	114	109	109
query41	44	43	43	43
query42	118	99	101	99
query43	466	410	411	410
query44	1186	752	750	750
query45	209	181	173	173
query46	1100	804	792	792
query47	1820	1720	1747	1720
query48	363	300	293	293
query49	1004	430	419	419
query50	908	431	430	430
query51	6766	6629	6586	6586
query52	96	93	85	85
query53	253	187	182	182
query54	622	451	452	451
query55	78	77	75	75
query56	291	262	252	252
query57	1146	1045	1033	1033
query58	269	282	265	265
query59	2498	2233	2290	2233
query60	325	269	276	269
query61	95	94	92	92
query62	872	676	656	656
query63	213	179	176	176
query64	5704	1911	1863	1863
query65	3152	3109	3073	3073
query66	1055	331	327	327
query67	15392	14947	14837	14837
query68	4344	565	576	565
query69	443	301	299	299
query70	1117	1070	1021	1021
query71	417	285	275	275
query72	7066	2658	2527	2527
query73	768	336	331	331
query74	6069	5647	5680	5647
query75	3380	2712	2749	2712
query76	2058	1230	1282	1230
query77	466	314	308	308
query78	9362	8953	8920	8920
query79	2384	546	539	539
query80	940	505	498	498
query81	560	224	223	223
query82	1009	137	127	127
query83	252	172	167	167
query84	264	81	78	78
query85	1199	314	297	297
query86	458	351	303	303
query87	3305	3094	3097	3094
query88	3209	2540	2522	2522
query89	388	289	287	287
query90	1693	188	192	188
query91	124	99	98	98
query92	59	50	48	48
query93	1910	626	632	626
query94	789	293	296	293
query95	382	268	263	263
query96	620	289	285	285
query97	3248	3026	3106	3026
query98	220	203	197	197
query99	1639	1292	1257	1257
Total cold run time: 260887 ms
Total hot run time: 169641 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.75 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8ad86a02268c559406641ad36468b907115bce7c, data reload: false

query1	0.04	0.04	0.04
query2	0.08	0.05	0.04
query3	0.22	0.05	0.05
query4	1.69	0.06	0.06
query5	0.48	0.48	0.47
query6	1.14	0.72	0.72
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.57	0.51	0.50
query10	0.55	0.57	0.56
query11	0.15	0.11	0.12
query12	0.15	0.12	0.12
query13	0.62	0.60	0.60
query14	0.77	0.79	0.79
query15	0.90	0.86	0.87
query16	0.36	0.35	0.36
query17	1.00	1.02	0.99
query18	0.21	0.21	0.21
query19	1.83	1.74	1.71
query20	0.02	0.01	0.01
query21	15.40	0.75	0.64
query22	3.97	7.28	1.18
query23	18.03	1.33	1.27
query24	2.28	0.23	0.21
query25	0.19	0.08	0.08
query26	0.31	0.21	0.21
query27	0.46	0.22	0.23
query28	13.20	1.00	0.98
query29	12.61	3.32	3.28
query30	0.25	0.05	0.06
query31	2.86	0.41	0.40
query32	3.25	0.48	0.49
query33	2.97	2.91	2.91
query34	15.48	4.26	4.26
query35	4.29	4.28	4.26
query36	0.69	0.48	0.48
query37	0.19	0.16	0.16
query38	0.15	0.16	0.16
query39	0.04	0.03	0.04
query40	0.17	0.13	0.12
query41	0.11	0.05	0.05
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 107.85 s
Total hot run time: 29.75 s

@gavinchou gavinchou changed the title [chore](meta) Add more log for FE meta [chore](meta) Add more log for FE meta and revert #37293 Aug 4, 2024
@gavinchou
Copy link
Contributor Author

run cloud_p0

dataroaring
dataroaring approved these changes Aug 4, 2024
View reviewed changes
Copy link
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 4, 2024
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 4, 2024

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Aug 4, 2024

PR approved by anyone and no changes requested.

@gavinchou gavinchou changed the title [chore](meta) Add more log for FE meta and revert #37293 Revert "(cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" Aug 4, 2024
@dataroaring dataroaring changed the title Revert "(cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" [revert](cloud-merge) Support shadow tablet to do cumulative compaction in cloud mode (apache#37293)" Aug 4, 2024
@dataroaring dataroaring merged commit 309f3b6 into apache:master Aug 4, 2024
32 of 35 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@github-actions github-actions[bot] github-actions[bot] left review comments

@dataroaring dataroaring dataroaring approved these changes

@Yukang-Lian Yukang-Lian Yukang-Lian approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. doing meta-change reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@gavinchou @doris-robot @Yukang-Lian @dataroaring