Skip to content

[feature](binlog) Wrap rename table/column binlog in BarrierLog#43033

Merged
dataroaring merged 2 commits intoapache:branch-2.0from
w41ter:support_rename_table_in_2.0
Nov 1, 2024
Merged

[feature](binlog) Wrap rename table/column binlog in BarrierLog#43033
dataroaring merged 2 commits intoapache:branch-2.0from
w41ter:support_rename_table_in_2.0

Conversation

@w41ter
Copy link
Contributor

@w41ter w41ter commented Oct 31, 2024

This PR is part of #37031, #39782

@doris-robot
Copy link

doris-robot commented Oct 31, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@w41ter w41ter force-pushed the support_rename_table_in_2.0 branch from b600220 to 6c57fa0 Compare October 31, 2024 15:01
@w41ter
Copy link
Contributor Author

w41ter commented Oct 31, 2024

run buildall

@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

1 similar comment
@github-actions
Copy link
Contributor

github-actions bot commented Oct 31, 2024

clang-tidy review says "All clean, LGTM! 👍"

@doris-robot
Copy link

doris-robot commented Oct 31, 2024

TPC-H: Total hot run time: 49323 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 6c57fa07bea9c760e6a2d9af950147bc525799ec, data reload: false

------ Round 1 ----------------------------------
q1	17753	4470	4323	4323
q2	2083	151	149	149
q3	10449	1923	1938	1923
q4	10301	1277	1317	1277
q5	8499	3857	3964	3857
q6	233	125	124	124
q7	2023	1610	1596	1596
q8	9339	2753	2730	2730
q9	10537	10329	10009	10009
q10	8660	3588	3521	3521
q11	418	249	253	249
q12	469	293	290	290
q13	18347	3987	4080	3987
q14	356	330	327	327
q15	504	460	456	456
q16	544	458	451	451
q17	1135	953	922	922
q18	7230	6913	6831	6831
q19	1699	1602	1513	1513
q20	548	318	307	307
q21	4423	4187	4093	4093
q22	491	406	388	388
Total cold run time: 116041 ms
Total hot run time: 49323 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4306	4284	4274	4274
q2	326	228	224	224
q3	4157	4094	4156	4094
q4	2753	2760	2745	2745
q5	7216	7164	7088	7088
q6	242	121	118	118
q7	3227	2828	2789	2789
q8	4377	4458	4497	4458
q9	14222	14095	13996	13996
q10	4255	4282	4262	4262
q11	747	684	679	679
q12	1034	860	866	860
q13	6690	3741	3745	3741
q14	462	433	420	420
q15	500	463	456	456
q16	629	582	589	582
q17	3969	3816	3799	3799
q18	8734	8756	8656	8656
q19	1733	1654	1664	1654
q20	2393	2127	2098	2098
q21	8455	8502	8460	8460
q22	1009	938	979	938
Total cold run time: 81436 ms
Total hot run time: 76391 ms

@doris-robot
Copy link

doris-robot commented Oct 31, 2024

TPC-DS: Total hot run time: 212558 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 6c57fa07bea9c760e6a2d9af950147bc525799ec, data reload: false

query1	942	390	412	390
query2	6527	2228	2115	2115
query3	6924	196	200	196
query4	23242	21400	21169	21169
query5	19721	6525	6568	6525
query6	297	216	252	216
query7	4164	291	306	291
query8	254	254	255	254
query9	3020	2629	2547	2547
query10	418	314	323	314
query11	15880	15029	15382	15029
query12	124	80	73	73
query13	1013	440	429	429
query14	17223	13851	13589	13589
query15	373	212	236	212
query16	6479	287	260	260
query17	1731	940	895	895
query18	893	311	313	311
query19	213	145	150	145
query20	106	102	94	94
query21	190	96	101	96
query22	5271	5038	5055	5038
query23	34301	33395	33425	33395
query24	7697	6319	6337	6319
query25	524	419	428	419
query26	1261	162	172	162
query27	2383	289	288	288
query28	6091	2226	2189	2189
query29	2791	2656	2600	2600
query30	245	170	169	169
query31	985	741	762	741
query32	73	63	58	58
query33	448	260	248	248
query34	859	475	462	462
query35	1149	918	957	918
query36	1161	1110	1125	1110
query37	91	60	61	60
query38	3076	2914	2916	2914
query39	1372	1326	1314	1314
query40	303	97	96	96
query41	47	39	36	36
query42	88	90	85	85
query43	605	592	630	592
query44	1168	710	722	710
query45	245	231	226	226
query46	1228	940	969	940
query47	1772	1669	1822	1669
query48	515	419	413	413
query49	646	376	381	376
query50	863	585	596	585
query51	4787	4686	4691	4686
query52	87	80	83	80
query53	223	178	195	178
query54	2661	2493	2468	2468
query55	91	81	87	81
query56	227	210	205	205
query57	1307	1146	1256	1146
query58	219	213	211	211
query59	3474	3332	3190	3190
query60	212	217	203	203
query61	98	96	96	96
query62	855	448	512	448
query63	202	176	174	174
query64	3466	1552	1447	1447
query65	3584	3528	3569	3528
query66	766	431	411	411
query67	16093	15813	15802	15802
query68	9757	644	657	644
query69	502	270	270	270
query70	1490	1387	1304	1304
query71	418	311	309	309
query72	6864	4799	4846	4799
query73	756	311	321	311
query74	6317	5805	5875	5805
query75	5324	3679	3673	3673
query76	5427	1119	1131	1119
query77	926	256	256	256
query78	12800	11633	11676	11633
query79	6410	657	682	657
query80	1734	397	402	397
query81	491	238	235	235
query82	1710	100	99	99
query83	172	130	134	130
query84	257	69	71	69
query85	891	334	321	321
query86	331	282	293	282
query87	3187	2989	3011	2989
query88	4675	2298	2303	2298
query89	420	288	305	288
query90	2027	222	222	222
query91	170	126	133	126
query92	58	50	53	50
query93	5624	578	547	547
query94	700	213	204	204
query95	2053	2060	2053	2053
query96	644	326	318	318
query97	6441	6444	6439	6439
query98	231	206	201	201
query99	2861	874	920	874
Total cold run time: 317012 ms
Total hot run time: 212558 ms

@doris-robot
Copy link

doris-robot commented Oct 31, 2024

ClickBench: Total hot run time: 31.37 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 6c57fa07bea9c760e6a2d9af950147bc525799ec, data reload: false

query1	0.02	0.02	0.02
query2	0.07	0.02	0.02
query3	0.24	0.05	0.05
query4	1.78	0.08	0.07
query5	0.53	0.53	0.52
query6	1.23	0.60	0.60
query7	0.02	0.01	0.02
query8	0.03	0.02	0.02
query9	0.52	0.50	0.49
query10	0.53	0.54	0.53
query11	0.12	0.09	0.08
query12	0.11	0.09	0.08
query13	0.61	0.62	0.61
query14	0.77	0.80	0.79
query15	0.78	0.75	0.76
query16	0.38	0.37	0.37
query17	0.99	1.03	1.00
query18	0.22	0.28	0.23
query19	1.89	1.85	1.88
query20	0.02	0.01	0.00
query21	15.48	0.55	0.54
query22	2.05	2.64	2.38
query23	17.18	0.97	0.88
query24	3.65	0.89	1.53
query25	0.40	0.08	0.04
query26	0.44	0.15	0.15
query27	0.04	0.05	0.04
query28	9.07	0.71	0.77
query29	12.60	2.29	2.26
query30	0.64	0.59	0.53
query31	2.81	0.39	0.37
query32	3.39	0.50	0.50
query33	3.05	3.11	3.06
query34	15.25	4.82	4.83
query35	4.90	4.87	4.84
query36	1.05	1.02	1.03
query37	0.06	0.05	0.04
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.17	0.14	0.14
query41	0.06	0.02	0.01
query42	0.03	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 103.26 s
Total hot run time: 31.37 s

@doris-robot
Copy link

doris-robot commented Oct 31, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 6c57fa07bea9c760e6a2d9af950147bc525799ec with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          58 seconds loaded 1101869774 Bytes, about 18 MB/s
Stream load parquet:      32 seconds loaded 861443392 Bytes, about 25 MB/s
Insert into select:       21.3 seconds inserted 10000000 Rows, about 469K ops/s

@dataroaring dataroaring merged commit f3a8be6 into apache:branch-2.0 Nov 1, 2024
@w41ter w41ter deleted the support_rename_table_in_2.0 branch November 4, 2024 02:38
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 4, 2024
@w41ter
[feature](binlog) Wrap rename table/column binlog in BarrierLog (apac…
3ea64ad 
…he#43033)

This PR is part of apache#37031, apache#39782
@w41ter w41ter mentioned this pull request Nov 4, 2024
Merged
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 8, 2024
@w41ter
[feature](binlog) Wrap rename table/column binlog in BarrierLog (apac…
4a5f0d2 
…he#43033)

This PR is part of apache#37031, apache#39782
w41ter added a commit to w41ter/incubator-doris that referenced this pull request Nov 8, 2024
@w41ter
[feature](binlog) Wrap rename table/column binlog in BarrierLog (apac…
2062c48 
…he#43033)

This PR is part of apache#37031, apache#39782
dataroaring pushed a commit that referenced this pull request Nov 8, 2024
@w41ter
[feature](binlog) Wrap rename table/column binlog in BarrierLog #43033 (
8f2f4e9 
#43168)

cherry pick from #43033
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@w41ter @doris-robot @dataroaring