Skip to content

[feature](backup) backup restore colocate table #46059

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
liutang123 merged 6 commits into from
Jan 21, 2025
Merged

[feature](backup) backup restore colocate table #46059

liutang123 merged 6 commits into from
Jan 21, 2025

Conversation

@justfortaste
Copy link
Contributor

@justfortaste justfortaste commented Dec 26, 2024 •
edited
Loading

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen
Copy link
Contributor

hello-stephen commented Dec 26, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@justfortaste
Copy link
Contributor Author

justfortaste commented Dec 26, 2024

run buildall

1 similar comment
@justfortaste
Copy link
Contributor Author

justfortaste commented Dec 26, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Dec 26, 2024

TPC-H: Total hot run time: 32947 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit ad3cfd2f143eea6042e2685f7883319cad57252a, data reload: false

------ Round 1 ----------------------------------
q1	17588	6192	6170	6170
q2	2045	325	174	174
q3	10387	1356	713	713
q4	10192	916	449	449
q5	7532	2317	2034	2034
q6	221	182	147	147
q7	909	763	609	609
q8	9240	1484	1278	1278
q9	5300	4956	4987	4956
q10	6783	2318	1872	1872
q11	489	272	269	269
q12	351	374	229	229
q13	17752	3635	2980	2980
q14	237	236	206	206
q15	578	502	499	499
q16	632	639	579	579
q17	571	891	325	325
q18	7142	6452	6355	6355
q19	1226	1045	589	589
q20	322	323	182	182
q21	2976	2261	2020	2020
q22	368	323	312	312
Total cold run time: 102841 ms
Total hot run time: 32947 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6346	6388	6391	6388
q2	249	329	230	230
q3	2241	2656	2298	2298
q4	1432	1829	1359	1359
q5	4352	4788	4996	4788
q6	196	180	146	146
q7	2152	1985	1795	1795
q8	2661	2868	2713	2713
q9	7329	7252	7230	7230
q10	3077	3311	2853	2853
q11	590	522	496	496
q12	664	761	596	596
q13	3349	3810	3091	3091
q14	282	293	298	293
q15	593	513	503	503
q16	636	699	650	650
q17	1266	1748	1251	1251
q18	7727	7424	7289	7289
q19	848	1196	1146	1146
q20	1942	1969	1839	1839
q21	5446	5050	4885	4885
q22	613	612	581	581
Total cold run time: 53991 ms
Total hot run time: 52420 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2024

TPC-DS: Total hot run time: 188856 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit ad3cfd2f143eea6042e2685f7883319cad57252a, data reload: false

query1	958	373	362	362
query2	6518	2469	2405	2405
query3	6706	218	208	208
query4	34076	23662	23321	23321
query5	4293	630	468	468
query6	287	195	198	195
query7	4628	491	305	305
query8	282	240	233	233
query9	9532	2721	2703	2703
query10	436	317	235	235
query11	18086	15444	15549	15444
query12	165	106	106	106
query13	1679	558	408	408
query14	12108	6716	7202	6716
query15	237	194	182	182
query16	8095	577	433	433
query17	1534	745	546	546
query18	2095	390	284	284
query19	211	181	155	155
query20	116	111	109	109
query21	204	119	99	99
query22	4263	4538	4140	4140
query23	34383	33343	33283	33283
query24	6215	2306	2246	2246
query25	511	459	373	373
query26	771	269	150	150
query27	2027	464	337	337
query28	5387	2463	2406	2406
query29	655	532	401	401
query30	230	179	149	149
query31	998	921	805	805
query32	94	57	56	56
query33	519	366	285	285
query34	738	829	513	513
query35	802	813	772	772
query36	995	1038	943	943
query37	115	104	73	73
query38	4145	4106	4140	4106
query39	1476	1452	1419	1419
query40	208	116	106	106
query41	49	51	51	51
query42	125	102	101	101
query43	521	543	501	501
query44	1283	798	805	798
query45	183	175	167	167
query46	871	1034	651	651
query47	1892	1883	1864	1864
query48	378	397	335	335
query49	751	488	394	394
query50	654	642	378	378
query51	7228	7263	7054	7054
query52	95	99	91	91
query53	218	249	183	183
query54	466	474	397	397
query55	80	78	77	77
query56	238	250	218	218
query57	1221	1158	1118	1118
query58	239	220	229	220
query59	3047	3146	2926	2926
query60	274	262	248	248
query61	109	115	107	107
query62	829	846	750	750
query63	238	189	191	189
query64	3747	1028	621	621
query65	3270	3128	3261	3128
query66	842	417	300	300
query67	16152	16164	15570	15570
query68	8645	744	507	507
query69	483	279	258	258
query70	1241	1127	1040	1040
query71	423	284	252	252
query72	5746	3794	3777	3777
query73	661	745	353	353
query74	9642	9379	8855	8855
query75	4568	3259	2614	2614
query76	4612	1200	776	776
query77	864	351	267	267
query78	10059	10251	9227	9227
query79	3071	895	591	591
query80	650	504	444	444
query81	488	269	238	238
query82	690	148	119	119
query83	164	172	153	153
query84	236	89	86	86
query85	805	362	301	301
query86	344	318	339	318
query87	4595	4605	4308	4308
query88	4023	2254	2207	2207
query89	410	326	305	305
query90	1937	186	186	186
query91	137	129	111	111
query92	65	56	50	50
query93	952	854	522	522
query94	657	373	276	276
query95	327	262	251	251
query96	497	606	279	279
query97	2749	2822	2642	2642
query98	231	234	199	199
query99	2561	1570	1427	1427
Total cold run time: 294311 ms
Total hot run time: 188856 ms

@doris-robot
Copy link

doris-robot commented Dec 26, 2024

ClickBench: Total hot run time: 31.98 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit ad3cfd2f143eea6042e2685f7883319cad57252a, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.03
query3	0.22	0.08	0.07
query4	1.61	0.10	0.10
query5	0.45	0.41	0.40
query6	1.13	0.65	0.66
query7	0.03	0.02	0.01
query8	0.04	0.03	0.02
query9	0.57	0.51	0.51
query10	0.55	0.55	0.55
query11	0.15	0.11	0.11
query12	0.14	0.11	0.12
query13	0.60	0.60	0.60
query14	2.72	2.77	2.76
query15	0.89	0.83	0.84
query16	0.38	0.39	0.39
query17	1.07	1.05	1.03
query18	0.23	0.21	0.21
query19	1.91	1.89	1.98
query20	0.01	0.02	0.01
query21	15.38	0.94	0.56
query22	0.76	0.89	0.57
query23	15.30	1.40	0.52
query24	3.24	2.01	1.68
query25	0.20	0.10	0.13
query26	0.28	0.17	0.14
query27	0.06	0.04	0.05
query28	14.38	1.45	1.05
query29	12.55	3.89	3.24
query30	0.24	0.09	0.06
query31	2.83	0.59	0.38
query32	3.24	0.55	0.46
query33	3.05	3.08	3.11
query34	16.76	5.11	4.50
query35	4.49	4.47	4.46
query36	0.66	0.49	0.49
query37	0.09	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.03	0.03
Total cold run time: 106.74 s
Total hot run time: 31.98 s

@justfortaste
Copy link
Contributor Author

justfortaste commented Jan 1, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 1, 2025

TPC-H: Total hot run time: 32874 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit deb4a6f4fa93212b7d8bbf1489a80c31e369c462, data reload: false

------ Round 1 ----------------------------------
q1	17968	6270	6096	6096
q2	2056	319	171	171
q3	10674	1272	745	745
q4	10282	867	442	442
q5	8743	2187	1998	1998
q6	207	180	153	153
q7	882	755	617	617
q8	9219	1343	1139	1139
q9	5225	4962	4941	4941
q10	6763	2306	1881	1881
q11	462	265	258	258
q12	360	376	226	226
q13	17765	3602	2940	2940
q14	245	228	216	216
q15	558	506	491	491
q16	645	633	586	586
q17	590	855	327	327
q18	7165	6527	6467	6467
q19	1938	968	583	583
q20	315	313	188	188
q21	3029	2268	2085	2085
q22	367	333	324	324
Total cold run time: 105458 ms
Total hot run time: 32874 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6346	6207	6213	6207
q2	239	324	229	229
q3	2261	2677	2338	2338
q4	1471	1867	1425	1425
q5	4378	4783	5120	4783
q6	191	172	142	142
q7	2167	1926	1787	1787
q8	2688	2785	2707	2707
q9	7270	7219	7194	7194
q10	3067	3352	2810	2810
q11	588	521	491	491
q12	676	742	604	604
q13	3352	3765	3176	3176
q14	300	303	286	286
q15	567	510	520	510
q16	629	698	641	641
q17	1203	1698	1250	1250
q18	7626	7493	7159	7159
q19	825	1126	1046	1046
q20	1917	1949	1836	1836
q21	5608	5020	4918	4918
q22	636	653	573	573
Total cold run time: 54005 ms
Total hot run time: 52112 ms

@doris-robot
Copy link

doris-robot commented Jan 1, 2025

TPC-DS: Total hot run time: 192246 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit deb4a6f4fa93212b7d8bbf1489a80c31e369c462, data reload: false

query1	971	378	384	378
query2	6527	2363	2247	2247
query3	6707	214	208	208
query4	34089	24071	23948	23948
query5	4343	609	452	452
query6	284	194	193	193
query7	4628	484	308	308
query8	292	239	220	220
query9	9489	2721	2710	2710
query10	459	302	243	243
query11	18025	15393	15364	15364
query12	153	105	102	102
query13	1668	536	404	404
query14	10126	7528	7803	7528
query15	272	195	198	195
query16	8118	587	440	440
query17	1604	756	568	568
query18	2048	402	302	302
query19	213	189	159	159
query20	119	124	112	112
query21	212	123	104	104
query22	4402	4474	4379	4379
query23	36128	33677	33694	33677
query24	6610	2273	2242	2242
query25	448	433	390	390
query26	1089	246	146	146
query27	2090	462	339	339
query28	5489	2435	2393	2393
query29	535	513	414	414
query30	245	182	152	152
query31	981	890	783	783
query32	91	63	58	58
query33	506	335	288	288
query34	755	806	499	499
query35	792	841	717	717
query36	1021	1028	924	924
query37	121	100	79	79
query38	4124	4130	4034	4034
query39	1466	1442	1431	1431
query40	199	114	98	98
query41	47	44	45	44
query42	127	104	105	104
query43	507	511	497	497
query44	1300	792	795	792
query45	182	172	167	167
query46	882	1042	639	639
query47	1916	1936	1868	1868
query48	386	399	314	314
query49	774	471	396	396
query50	634	643	403	403
query51	7206	7266	7024	7024
query52	110	102	91	91
query53	217	247	180	180
query54	472	489	387	387
query55	83	78	80	78
query56	267	262	263	262
query57	1186	1181	1097	1097
query58	239	223	222	222
query59	3155	3154	2942	2942
query60	284	260	245	245
query61	127	110	110	110
query62	872	779	732	732
query63	229	188	205	188
query64	4467	1016	686	686
query65	3289	3159	3312	3159
query66	850	425	306	306
query67	15864	15849	15501	15501
query68	9871	746	517	517
query69	461	308	249	249
query70	1153	1112	1137	1112
query71	440	276	261	261
query72	5834	3834	3788	3788
query73	674	736	364	364
query74	9693	9273	9170	9170
query75	4469	3162	2678	2678
query76	5527	1192	771	771
query77	1054	365	292	292
query78	10681	10300	10487	10300
query79	2348	825	606	606
query80	728	515	419	419
query81	479	270	226	226
query82	462	151	122	122
query83	192	163	143	143
query84	281	93	73	73
query85	798	365	311	311
query86	356	334	299	299
query87	4458	4604	4373	4373
query88	3525	2226	2211	2211
query89	404	325	304	304
query90	1881	184	185	184
query91	140	133	110	110
query92	62	55	52	52
query93	983	792	535	535
query94	664	403	297	297
query95	328	260	254	254
query96	496	605	277	277
query97	2796	2808	2646	2646
query98	225	200	201	200
query99	1640	1576	1440	1440
Total cold run time: 296214 ms
Total hot run time: 192246 ms

@doris-robot
Copy link

doris-robot commented Jan 1, 2025

ClickBench: Total hot run time: 31.12 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit deb4a6f4fa93212b7d8bbf1489a80c31e369c462, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.04
query3	0.24	0.07	0.06
query4	1.61	0.10	0.11
query5	0.42	0.41	0.41
query6	1.18	0.65	0.65
query7	0.02	0.01	0.02
query8	0.04	0.03	0.03
query9	0.59	0.48	0.53
query10	0.54	0.56	0.56
query11	0.15	0.10	0.10
query12	0.14	0.11	0.12
query13	0.61	0.61	0.60
query14	2.88	2.87	2.87
query15	0.92	0.82	0.82
query16	0.38	0.40	0.38
query17	1.07	1.06	1.00
query18	0.23	0.21	0.21
query19	2.00	1.80	1.98
query20	0.01	0.01	0.01
query21	15.36	0.90	0.60
query22	0.77	0.97	0.62
query23	15.17	1.42	0.51
query24	3.00	1.83	0.85
query25	0.21	0.22	0.06
query26	0.20	0.14	0.14
query27	0.06	0.04	0.04
query28	14.11	1.52	1.05
query29	12.57	3.98	3.27
query30	0.24	0.08	0.06
query31	2.82	0.61	0.40
query32	3.23	0.55	0.46
query33	3.06	3.07	3.14
query34	16.94	5.15	4.50
query35	4.46	4.49	4.46
query36	0.64	0.50	0.48
query37	0.09	0.06	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.02
query40	0.17	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 106.47 s
Total hot run time: 31.12 s

@justfortaste justfortaste changed the title [feature](backup) backup restore coloate table [feature](backup) backup restore colocate table Jan 9, 2025
@justfortaste justfortaste force-pushed the feature_backup_colocate_master branch from deb4a6f to b3bd5e4 Compare January 16, 2025 10:13
@justfortaste
Copy link
Contributor Author

justfortaste commented Jan 17, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 17, 2025

TPC-H: Total hot run time: 31919 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 69878d2247d88e0459a3e0e0b9d9751a6b991272, data reload: false

------ Round 1 ----------------------------------
q1	17572	5419	5371	5371
q2	2048	289	168	168
q3	10494	1196	728	728
q4	10212	962	509	509
q5	7515	2354	2140	2140
q6	186	163	134	134
q7	878	757	602	602
q8	9235	1343	1139	1139
q9	5178	4757	4852	4757
q10	6818	2323	1900	1900
q11	484	268	259	259
q12	337	349	220	220
q13	17793	3695	3022	3022
q14	224	221	211	211
q15	521	474	457	457
q16	631	633	590	590
q17	558	845	322	322
q18	6898	6337	6425	6337
q19	2130	951	557	557
q20	306	321	190	190
q21	2770	2161	2006	2006
q22	364	347	300	300
Total cold run time: 103152 ms
Total hot run time: 31919 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5539	5432	5458	5432
q2	239	330	235	235
q3	2270	2600	2306	2306
q4	1380	1797	1354	1354
q5	4309	4732	4633	4633
q6	164	158	128	128
q7	1970	1952	1788	1788
q8	2626	2776	2679	2679
q9	7349	7243	7204	7204
q10	3005	3299	2776	2776
q11	594	510	507	507
q12	693	737	587	587
q13	3431	3878	3317	3317
q14	290	301	268	268
q15	517	469	474	469
q16	662	699	670	670
q17	1218	1755	1279	1279
q18	7669	7431	7341	7341
q19	775	777	1140	777
q20	2004	2053	1867	1867
q21	5534	5177	4804	4804
q22	603	607	568	568
Total cold run time: 52841 ms
Total hot run time: 50989 ms

@doris-robot
Copy link

doris-robot commented Jan 17, 2025

TPC-DS: Total hot run time: 195004 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 69878d2247d88e0459a3e0e0b9d9751a6b991272, data reload: false

query1	1322	976	914	914
query2	6400	2110	2112	2110
query3	10969	4477	4311	4311
query4	60959	32349	23223	23223
query5	5104	609	451	451
query6	433	184	170	170
query7	5561	497	304	304
query8	325	239	227	227
query9	8364	2570	2568	2568
query10	434	311	254	254
query11	15944	15209	15331	15209
query12	158	109	109	109
query13	1454	540	430	430
query14	10954	7896	7078	7078
query15	217	218	209	209
query16	6691	651	484	484
query17	1126	724	594	594
query18	1031	400	321	321
query19	204	208	170	170
query20	118	118	116	116
query21	217	123	110	110
query22	4658	4840	4338	4338
query23	34235	33431	33749	33431
query24	5768	2284	2376	2284
query25	447	481	401	401
query26	642	279	154	154
query27	1785	469	335	335
query28	4094	2441	2437	2437
query29	542	628	422	422
query30	220	187	162	162
query31	944	872	849	849
query32	71	62	59	59
query33	431	377	329	329
query34	747	867	521	521
query35	831	834	800	800
query36	1047	1066	963	963
query37	126	99	81	81
query38	4330	4369	4231	4231
query39	1517	1471	1433	1433
query40	203	122	107	107
query41	53	50	51	50
query42	123	109	109	109
query43	543	549	515	515
query44	1334	841	822	822
query45	181	180	167	167
query46	871	1081	695	695
query47	1903	1912	1866	1866
query48	387	417	335	335
query49	712	495	416	416
query50	666	681	425	425
query51	6987	7013	6898	6898
query52	104	104	100	100
query53	237	259	189	189
query54	484	503	427	427
query55	84	85	87	85
query56	268	270	273	270
query57	1192	1210	1158	1158
query58	242	233	235	233
query59	3061	3169	3027	3027
query60	272	289	256	256
query61	151	118	118	118
query62	738	700	644	644
query63	217	189	193	189
query64	1287	1037	653	653
query65	3246	3197	3155	3155
query66	710	397	298	298
query67	16098	15848	15514	15514
query68	5070	813	541	541
query69	497	288	263	263
query70	1228	1157	1161	1157
query71	417	277	258	258
query72	6369	3822	3878	3822
query73	812	744	364	364
query74	9945	9113	9076	9076
query75	3199	3176	2673	2673
query76	3811	1176	779	779
query77	510	376	284	284
query78	10155	10076	9285	9285
query79	2473	800	596	596
query80	889	543	445	445
query81	524	273	238	238
query82	356	147	128	128
query83	215	172	152	152
query84	292	97	83	83
query85	755	373	319	319
query86	421	311	306	306
query87	4441	4543	4516	4516
query88	3796	2150	2174	2150
query89	399	328	287	287
query90	1633	194	193	193
query91	139	153	111	111
query92	68	56	53	53
query93	2738	865	521	521
query94	751	404	307	307
query95	339	277	257	257
query96	490	608	303	303
query97	2764	2877	2774	2774
query98	220	199	193	193
query99	1289	1391	1257	1257
Total cold run time: 309779 ms
Total hot run time: 195004 ms

@doris-robot
Copy link

doris-robot commented Jan 17, 2025

ClickBench: Total hot run time: 30.55 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 69878d2247d88e0459a3e0e0b9d9751a6b991272, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.07	0.07
query4	1.62	0.10	0.11
query5	0.41	0.42	0.41
query6	1.17	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.58	0.51	0.50
query10	0.54	0.57	0.54
query11	0.15	0.10	0.10
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	2.72	2.74	2.83
query15	0.91	0.85	0.85
query16	0.39	0.39	0.41
query17	1.00	1.09	0.99
query18	0.22	0.22	0.21
query19	1.99	1.81	1.97
query20	0.02	0.01	0.01
query21	15.36	0.96	0.60
query22	0.76	0.95	0.66
query23	15.19	1.48	0.56
query24	3.73	0.78	1.61
query25	0.14	0.13	0.15
query26	0.37	0.15	0.14
query27	0.06	0.06	0.04
query28	12.94	0.99	0.44
query29	12.57	4.06	3.38
query30	0.24	0.09	0.07
query31	2.82	0.58	0.38
query32	3.23	0.55	0.46
query33	3.00	3.06	3.01
query34	16.53	5.12	4.40
query35	4.50	4.50	4.52
query36	0.65	0.48	0.50
query37	0.10	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.02
query40	0.17	0.14	0.13
query41	0.08	0.03	0.03
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.45 s
Total hot run time: 30.55 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2025

PR approved by at least one committer and no changes requested.

@github-actions
Copy link
Contributor

github-actions bot commented Jan 17, 2025

PR approved by anyone and no changes requested.

lide-reed
lide-reed approved these changes Jan 20, 2025
View reviewed changes
Copy link
Contributor

@lide-reed lide-reed left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lide-reed
Copy link
Contributor

lide-reed commented Jan 20, 2025

run buildall

@justfortaste
Copy link
Contributor Author

justfortaste commented Jan 20, 2025

run p0

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-H: Total hot run time: 32168 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit df90ff7acba8ccbf0a51d92aff03c8dc183c3c5c, data reload: false

------ Round 1 ----------------------------------
q1	17586	5483	5376	5376
q2	2055	318	170	170
q3	10397	1280	702	702
q4	10201	969	516	516
q5	7515	2384	2161	2161
q6	191	165	132	132
q7	889	767	632	632
q8	9260	1381	1119	1119
q9	5374	4926	4894	4894
q10	6826	2324	1867	1867
q11	464	266	268	266
q12	351	357	222	222
q13	17768	3646	3114	3114
q14	229	239	209	209
q15	510	482	469	469
q16	640	603	596	596
q17	543	860	330	330
q18	6837	6535	6371	6371
q19	1211	959	532	532
q20	315	322	192	192
q21	2806	2132	1985	1985
q22	369	333	313	313
Total cold run time: 102337 ms
Total hot run time: 32168 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5439	5396	5456	5396
q2	234	333	230	230
q3	2267	2684	2317	2317
q4	1433	1841	1419	1419
q5	4307	4729	4628	4628
q6	160	157	125	125
q7	1983	2041	1842	1842
q8	2646	2830	2691	2691
q9	7306	7305	7280	7280
q10	2976	3243	2788	2788
q11	586	509	519	509
q12	662	774	614	614
q13	3525	3902	3281	3281
q14	276	294	275	275
q15	511	482	460	460
q16	644	667	668	667
q17	1214	1725	1260	1260
q18	7726	7540	7203	7203
q19	771	1109	1057	1057
q20	1971	2036	1856	1856
q21	5541	5284	4987	4987
q22	611	610	597	597
Total cold run time: 52789 ms
Total hot run time: 51482 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-DS: Total hot run time: 187455 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit df90ff7acba8ccbf0a51d92aff03c8dc183c3c5c, data reload: false

query1	978	379	369	369
query2	6538	2103	2083	2083
query3	6804	213	212	212
query4	33061	23387	23132	23132
query5	4384	606	482	482
query6	278	196	188	188
query7	4615	480	308	308
query8	277	241	218	218
query9	9588	2564	2565	2564
query10	483	314	263	263
query11	18304	15241	15141	15141
query12	154	113	104	104
query13	1692	503	380	380
query14	10322	6857	7443	6857
query15	254	194	201	194
query16	8403	601	419	419
query17	1617	731	552	552
query18	2098	389	303	303
query19	220	172	157	157
query20	113	119	114	114
query21	203	126	103	103
query22	4255	4468	4232	4232
query23	33993	32987	32979	32979
query24	6508	2249	2320	2249
query25	484	450	378	378
query26	752	269	159	159
query27	2021	471	329	329
query28	5232	2407	2391	2391
query29	648	545	419	419
query30	229	191	161	161
query31	986	863	782	782
query32	93	63	59	59
query33	501	353	297	297
query34	761	857	490	490
query35	813	812	725	725
query36	998	1027	932	932
query37	118	98	74	74
query38	4158	4173	4083	4083
query39	1423	1380	1409	1380
query40	207	121	100	100
query41	54	50	53	50
query42	126	103	101	101
query43	516	524	482	482
query44	1274	791	798	791
query45	185	176	169	169
query46	841	1033	634	634
query47	1812	1836	1795	1795
query48	377	386	311	311
query49	743	480	409	409
query50	619	642	397	397
query51	6844	6993	6802	6802
query52	101	106	89	89
query53	218	247	191	191
query54	478	469	409	409
query55	81	79	76	76
query56	267	272	227	227
query57	1161	1165	1091	1091
query58	249	231	254	231
query59	2940	3259	2966	2966
query60	271	267	273	267
query61	113	112	118	112
query62	797	722	670	670
query63	218	188	185	185
query64	3467	1030	653	653
query65	3269	3158	3146	3146
query66	883	416	308	308
query67	15622	15354	15309	15309
query68	2228	820	543	543
query69	418	300	251	251
query70	1155	1184	1117	1117
query71	316	299	253	253
query72	6160	3975	3993	3975
query73	618	742	351	351
query74	10031	8910	8776	8776
query75	3118	3136	2654	2654
query76	2205	1114	779	779
query77	329	350	287	287
query78	10050	10073	9304	9304
query79	3173	775	609	609
query80	1783	529	449	449
query81	559	275	239	239
query82	412	147	119	119
query83	262	176	149	149
query84	238	95	71	71
query85	793	352	284	284
query86	476	317	310	310
query87	4455	4402	4406	4402
query88	4431	2145	2106	2106
query89	388	328	294	294
query90	1926	192	194	192
query91	130	134	107	107
query92	69	57	53	53
query93	2411	846	534	534
query94	735	402	292	292
query95	335	273	291	273
query96	473	607	281	281
query97	2832	2801	2713	2713
query98	234	205	197	197
query99	1295	1392	1258	1258
Total cold run time: 282130 ms
Total hot run time: 187455 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

ClickBench: Total hot run time: 30.72 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit df90ff7acba8ccbf0a51d92aff03c8dc183c3c5c, data reload: false

query1	0.03	0.03	0.02
query2	0.07	0.03	0.03
query3	0.24	0.08	0.06
query4	1.61	0.11	0.10
query5	0.41	0.41	0.40
query6	1.15	0.64	0.65
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.59	0.50	0.48
query10	0.57	0.56	0.55
query11	0.14	0.10	0.09
query12	0.14	0.10	0.11
query13	0.62	0.61	0.59
query14	2.81	2.76	2.72
query15	0.89	0.82	0.84
query16	0.38	0.37	0.38
query17	0.99	1.04	1.07
query18	0.22	0.21	0.21
query19	1.99	1.86	2.02
query20	0.02	0.01	0.01
query21	15.36	0.91	0.58
query22	0.74	0.79	0.84
query23	15.11	1.43	0.59
query24	3.03	0.97	1.66
query25	0.13	0.12	0.12
query26	0.32	0.15	0.14
query27	0.07	0.06	0.05
query28	14.07	0.98	0.42
query29	12.57	3.92	3.26
query30	0.25	0.09	0.07
query31	2.82	0.63	0.40
query32	3.23	0.54	0.46
query33	2.98	2.99	3.01
query34	16.42	5.18	4.49
query35	4.54	4.51	4.47
query36	0.65	0.51	0.48
query37	0.10	0.07	0.06
query38	0.05	0.04	0.03
query39	0.03	0.02	0.03
query40	0.17	0.13	0.13
query41	0.08	0.03	0.02
query42	0.04	0.03	0.02
query43	0.04	0.03	0.02
Total cold run time: 105.73 s
Total hot run time: 30.72 s

@justfortaste
Copy link
Contributor Author

justfortaste commented Jan 20, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-H: Total hot run time: 31948 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7c9f9d4f1dcbdc68316a0df491d3ffb161be917e, data reload: false

------ Round 1 ----------------------------------
q1	17610	5533	5360	5360
q2	2053	309	174	174
q3	10404	1238	711	711
q4	10210	972	520	520
q5	7544	2382	2144	2144
q6	191	166	132	132
q7	896	752	608	608
q8	9242	1373	1144	1144
q9	5140	4890	4898	4890
q10	6846	2345	1887	1887
q11	467	268	262	262
q12	343	362	225	225
q13	17774	3626	3067	3067
q14	219	232	205	205
q15	512	474	472	472
q16	629	619	592	592
q17	567	863	336	336
q18	7070	6392	6257	6257
q19	4347	954	518	518
q20	304	310	185	185
q21	3012	2205	1950	1950
q22	361	329	309	309
Total cold run time: 105741 ms
Total hot run time: 31948 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5690	5544	5499	5499
q2	239	328	235	235
q3	2229	2609	2325	2325
q4	1389	1795	1332	1332
q5	4305	4736	4599	4599
q6	164	157	124	124
q7	1959	1846	1705	1705
q8	2496	2675	2600	2600
q9	6951	6929	6919	6919
q10	2853	3154	2651	2651
q11	566	490	487	487
q12	674	729	609	609
q13	3321	3657	3067	3067
q14	265	298	254	254
q15	500	475	461	461
q16	635	673	640	640
q17	1188	1691	1221	1221
q18	7344	7080	6865	6865
q19	751	1088	1054	1054
q20	1923	1950	1830	1830
q21	5485	5119	4702	4702
q22	594	617	555	555
Total cold run time: 51521 ms
Total hot run time: 49734 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

TPC-DS: Total hot run time: 187577 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7c9f9d4f1dcbdc68316a0df491d3ffb161be917e, data reload: false

query1	993	381	356	356
query2	6515	2149	2108	2108
query3	6796	212	209	209
query4	33400	23763	23246	23246
query5	4271	615	443	443
query6	270	196	182	182
query7	4593	478	306	306
query8	276	222	212	212
query9	9408	2573	2574	2573
query10	464	309	251	251
query11	17860	15076	15020	15020
query12	146	110	108	108
query13	1651	514	377	377
query14	9884	7232	6411	6411
query15	243	193	197	193
query16	7408	625	484	484
query17	1624	721	578	578
query18	1977	391	302	302
query19	210	196	184	184
query20	114	112	105	105
query21	212	124	102	102
query22	4174	4422	4284	4284
query23	34262	32754	33441	32754
query24	6832	2210	2268	2210
query25	472	447	383	383
query26	1190	268	153	153
query27	2099	466	326	326
query28	5265	2436	2393	2393
query29	579	546	434	434
query30	238	184	171	171
query31	951	891	820	820
query32	88	62	62	62
query33	510	346	296	296
query34	727	836	503	503
query35	790	800	762	762
query36	995	1020	950	950
query37	120	97	79	79
query38	4214	4248	4123	4123
query39	1462	1410	1375	1375
query40	201	112	101	101
query41	52	52	52	52
query42	131	108	100	100
query43	517	537	508	508
query44	1288	796	792	792
query45	179	173	162	162
query46	855	1038	634	634
query47	1842	1838	1799	1799
query48	387	399	317	317
query49	777	487	396	396
query50	630	652	401	401
query51	7006	6963	6800	6800
query52	103	99	88	88
query53	222	259	180	180
query54	484	491	406	406
query55	82	80	81	80
query56	272	275	257	257
query57	1195	1165	1124	1124
query58	252	247	250	247
query59	3123	3182	3020	3020
query60	284	273	278	273
query61	161	142	138	138
query62	799	719	671	671
query63	229	193	188	188
query64	4593	997	640	640
query65	3219	3200	3148	3148
query66	1084	416	306	306
query67	15976	15734	15422	15422
query68	4412	824	526	526
query69	456	294	280	280
query70	1231	1167	1170	1167
query71	384	278	266	266
query72	5794	3833	3913	3833
query73	646	754	352	352
query74	10362	9184	8972	8972
query75	3176	3197	2651	2651
query76	3247	1221	755	755
query77	464	372	290	290
query78	10061	10006	9350	9350
query79	3118	822	600	600
query80	752	521	464	464
query81	519	283	243	243
query82	350	156	119	119
query83	176	173	150	150
query84	251	101	75	75
query85	753	364	308	308
query86	405	324	302	302
query87	4417	4474	4507	4474
query88	5156	2132	2112	2112
query89	390	338	292	292
query90	1872	191	187	187
query91	136	134	110	110
query92	70	60	61	60
query93	2785	868	518	518
query94	760	398	286	286
query95	347	279	255	255
query96	494	618	279	279
query97	2822	2827	2740	2740
query98	229	204	203	203
query99	1282	1372	1280	1280
Total cold run time: 286543 ms
Total hot run time: 187577 ms

@doris-robot
Copy link

doris-robot commented Jan 20, 2025

ClickBench: Total hot run time: 31.02 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7c9f9d4f1dcbdc68316a0df491d3ffb161be917e, data reload: false

query1	0.03	0.06	0.03
query2	0.06	0.04	0.03
query3	0.24	0.06	0.07
query4	1.62	0.11	0.10
query5	0.41	0.42	0.41
query6	1.16	0.66	0.66
query7	0.02	0.02	0.02
query8	0.04	0.03	0.03
query9	0.60	0.50	0.50
query10	0.58	0.56	0.56
query11	0.14	0.10	0.12
query12	0.15	0.11	0.11
query13	0.60	0.60	0.60
query14	2.73	2.83	2.76
query15	0.92	0.84	0.82
query16	0.40	0.39	0.37
query17	1.04	1.02	1.03
query18	0.23	0.20	0.21
query19	1.89	1.81	2.01
query20	0.02	0.01	0.02
query21	15.37	0.90	0.60
query22	0.74	0.95	0.74
query23	15.08	1.38	0.63
query24	3.10	1.30	1.25
query25	0.19	0.18	0.06
query26	0.29	0.15	0.14
query27	0.07	0.05	0.06
query28	14.00	1.00	0.42
query29	12.56	3.98	3.27
query30	0.25	0.12	0.05
query31	2.82	0.60	0.39
query32	3.22	0.54	0.46
query33	2.94	3.00	3.09
query34	16.57	5.17	4.52
query35	4.53	4.53	4.46
query36	0.64	0.48	0.48
query37	0.09	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.02	0.02
query40	0.16	0.14	0.12
query41	0.08	0.03	0.02
query42	0.04	0.02	0.03
query43	0.03	0.03	0.03
Total cold run time: 105.73 s
Total hot run time: 31.02 s

@liutang123 liutang123 merged commit f199abc into apache:master Jan 21, 2025
24 checks passed
justfortaste added a commit to justfortaste/doris that referenced this pull request Feb 12, 2025
@justfortaste
[feature](backup) backup restore colocate table (apache#46059)
46c84cc 
1. Add a property `reserve_colocate` when backup a table to record what group a table belongs to.
2. When restore the table, create or find the colocate group, and add the table to the colocate group.
3. Remove the table from the colocate group when restore failed.
@justfortaste justfortaste mentioned this pull request Feb 12, 2025
Closed
lzyy2024 pushed a commit to lzyy2024/doris that referenced this pull request Feb 21, 2025
@justfortaste @lide-reed @lzyy2024
[feature](backup) backup restore colocate table (apache#46059)
7964a64 
### What problem does this PR solve?
1. Add a property `reserve_colocate` when backup a table to record what group a table belongs to.
2. When restore the table, create or find the colocate group, and add the table to the colocate group.
3. Remove the table from the colocate group when restore failed.

---------

Co-authored-by: lide <lide@apache.org>
liutang123 pushed a commit to liutang123/doris that referenced this pull request Apr 29, 2025
@justfortaste
[Tencent][feature](backup) backup restore colocate table (apache#46059)
a57cba2 
1. Add a property `reserve_colocate` when backup a table to record what group a table belongs to.
2. When restore the table, create or find the colocate group, and add the table to the colocate group.
3. Remove the table from the colocate group when restore failed.
cambyzju pushed a commit to cambyzju/incubator-doris that referenced this pull request Jul 17, 2025
@justfortaste
[Tencent][feature](backup) backup restore colocate table (apache#46059)
1056681 
1. Add a property `reserve_colocate` when backup a table to record what group a table belongs to.
2. When restore the table, create or find the colocate group, and add the table to the colocate group.
3. Remove the table from the colocate group when restore failed.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lide-reed lide-reed lide-reed approved these changes

@liutang123 liutang123 liutang123 approved these changes

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@justfortaste @hello-stephen @doris-robot @lide-reed @liutang123