Skip to content

[opt](iceberg) support create branch/tag for iceberg #51727

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 9 commits into from
Jun 24, 2025
Merged

[opt](iceberg) support create branch/tag for iceberg #51727

merged 9 commits into from
Jun 24, 2025

Conversation

wuwenchi
Copy link
Contributor

@wuwenchi wuwenchi commented Jun 15, 2025 •
edited by morningman
Loading

What problem does this PR solve?

Related Issue: #48285

Problem Summary:

You can now create Iceberg branches and tags through Doris.

-- branch 
alter table tb1 create branch b1;
alter table tb1 create branch if not exists b1;
alter table tb1 create or replace branch b1;
alter table tb1 create or replace branch b1 AS OF VERSION <version>;
alter table tb1 create or replace branch b1 AS OF VERSION <version> RETAIN 1 HOURS;
-- CREATE b1 at snapshot 1234, retain b1 for 30 days, and retain the latest 30 days. The latest 3 snapshot snapshots, and 2 days worth of snapshots. 
alter table tb1 CREATE BRANCH b1 AS OF VERSION 1234 RETAIN 30 DAYS WITH SNAPSHOT RETENTION 3 SNAPSHOTS 2 DAYS;

-- tag
alter table tb1 create tag t1;
alter table tb1 create tag if not eists t1;
alter table tb1 create or replace tag t1;
alter table tb1 create or replace tag t1 AS OF VERSION <version>;
-- CREATE t1 at snapshot 1234 and retain it for 1 year. 
alter table tb1 create tag t1 AS OF VERSION 1234 RETAIN 365 DAYS

The supported time units include: DAYS, HOURS, MINUTES

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jun 15, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@wuwenchi
Copy link
Contributor Author

run buildall

2 similar comments
@wuwenchi
Copy link
Contributor Author

run buildall

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 35852 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 82cd5a7ab644e84cd713ed442c90cf66065a0d33, data reload: false

------ Round 1 ----------------------------------
q1	17587	5361	5118	5118
q2	1953	290	191	191
q3	10274	1315	794	794
q4	10226	1065	536	536
q5	7537	2477	2424	2424
q6	211	175	133	133
q7	961	740	611	611
q8	9321	1493	1217	1217
q9	6768	5287	5285	5285
q10	6908	2348	1929	1929
q11	525	311	321	311
q12	368	385	225	225
q13	17844	3885	3299	3299
q14	250	241	229	229
q15	580	485	497	485
q16	480	453	423	423
q17	697	931	416	416
q18	7762	7216	7148	7148
q19	1554	1088	627	627
q20	376	374	244	244
q21	4239	4074	3187	3187
q22	1086	1061	1020	1020
Total cold run time: 107507 ms
Total hot run time: 35852 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5323	5270	5281	5270
q2	252	334	223	223
q3	2178	2682	2313	2313
q4	1408	1824	1403	1403
q5	4327	4346	4418	4346
q6	263	178	128	128
q7	2102	1979	1771	1771
q8	2751	2764	2722	2722
q9	7325	7289	7061	7061
q10	3005	3283	2804	2804
q11	606	526	499	499
q12	750	801	659	659
q13	3602	3974	3361	3361
q14	295	308	278	278
q15	537	484	476	476
q16	465	511	470	470
q17	1286	1609	1415	1415
q18	7829	7633	7722	7633
q19	928	929	1132	929
q20	2131	2060	1993	1993
q21	4918	4453	4623	4453
q22	1131	1103	1081	1081
Total cold run time: 53412 ms
Total hot run time: 51288 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 197410 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 82cd5a7ab644e84cd713ed442c90cf66065a0d33, data reload: false

query1	1428	1012	992	992
query2	6188	2036	1980	1980
query3	11036	4608	4515	4515
query4	54379	25704	23467	23467
query5	5142	621	484	484
query6	366	230	211	211
query7	4925	553	327	327
query8	308	260	230	230
query9	5715	2986	2989	2986
query10	482	367	307	307
query11	15192	15585	14898	14898
query12	160	115	110	110
query13	1082	593	454	454
query14	10285	6492	6441	6441
query15	213	217	202	202
query16	7088	673	538	538
query17	1107	811	626	626
query18	1575	432	341	341
query19	230	234	184	184
query20	175	137	122	122
query21	214	137	112	112
query22	4726	4609	4473	4473
query23	34815	33749	33668	33668
query24	6719	2465	2513	2465
query25	488	495	431	431
query26	708	283	178	178
query27	2290	539	391	391
query28	3130	2479	2455	2455
query29	617	583	471	471
query30	284	239	208	208
query31	924	852	799	799
query32	72	71	66	66
query33	473	374	340	340
query34	851	917	614	614
query35	811	881	806	806
query36	1038	1073	952	952
query37	118	113	87	87
query38	4230	4365	4392	4365
query39	1530	1466	1463	1463
query40	219	124	120	120
query41	67	61	60	60
query42	137	125	126	125
query43	562	575	529	529
query44	1432	945	939	939
query45	184	186	166	166
query46	932	1095	703	703
query47	1966	1927	1856	1856
query48	417	451	351	351
query49	662	505	437	437
query50	707	744	449	449
query51	4325	4351	4285	4285
query52	123	123	114	114
query53	248	276	212	212
query54	656	645	583	583
query55	96	84	94	84
query56	339	351	319	319
query57	1215	1240	1168	1168
query58	295	297	284	284
query59	2962	2957	2860	2860
query60	380	345	343	343
query61	131	136	137	136
query62	760	749	698	698
query63	244	205	206	205
query64	1799	1107	734	734
query65	4269	4180	4139	4139
query66	729	402	321	321
query67	16321	15997	16054	15997
query68	6614	971	591	591
query69	562	320	290	290
query70	1269	1237	1206	1206
query71	515	373	332	332
query72	5890	4815	5029	4815
query73	1499	726	401	401
query74	8984	9129	8973	8973
query75	4186	3245	2737	2737
query76	4232	1199	776	776
query77	759	413	306	306
query78	10127	10187	9361	9361
query79	3286	857	604	604
query80	637	537	458	458
query81	473	266	216	216
query82	450	128	104	104
query83	318	261	254	254
query84	299	118	98	98
query85	791	369	321	321
query86	360	331	300	300
query87	4448	4465	4415	4415
query88	3690	2470	2421	2421
query89	417	329	298	298
query90	1924	212	216	212
query91	153	147	113	113
query92	83	65	59	59
query93	1543	1029	653	653
query94	689	418	323	323
query95	393	315	344	315
query96	544	594	297	297
query97	2746	2770	2676	2676
query98	253	216	212	212
query99	1474	1459	1297	1297
Total cold run time: 302899 ms
Total hot run time: 197410 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.14 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 82cd5a7ab644e84cd713ed442c90cf66065a0d33, data reload: false

query1	0.04	0.04	0.03
query2	0.07	0.03	0.05
query3	0.24	0.06	0.06
query4	1.61	0.10	0.10
query5	0.44	0.42	0.44
query6	1.16	0.67	0.68
query7	0.02	0.01	0.02
query8	0.05	0.04	0.03
query9	0.60	0.55	0.54
query10	0.57	0.60	0.57
query11	0.16	0.12	0.12
query12	0.16	0.13	0.12
query13	0.62	0.63	0.60
query14	0.80	0.82	0.81
query15	0.88	0.86	0.87
query16	0.40	0.38	0.39
query17	1.06	1.08	1.05
query18	0.24	0.22	0.22
query19	1.94	1.86	1.93
query20	0.01	0.01	0.01
query21	15.40	0.92	0.55
query22	0.78	1.29	0.75
query23	14.72	1.41	0.67
query24	6.95	0.68	0.51
query25	0.45	0.26	0.09
query26	0.54	0.18	0.16
query27	0.06	0.06	0.06
query28	9.37	0.96	0.44
query29	12.59	4.21	3.50
query30	0.27	0.10	0.07
query31	2.82	0.62	0.41
query32	3.24	0.57	0.48
query33	3.10	3.11	3.19
query34	15.78	5.16	4.48
query35	4.52	4.53	4.56
query36	0.67	0.51	0.48
query37	0.09	0.07	0.06
query38	0.06	0.05	0.05
query39	0.04	0.03	0.03
query40	0.16	0.16	0.14
query41	0.08	0.02	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 102.84 s
Total hot run time: 29.14 s

@wuwenchi wuwenchi force-pushed the opt-iceberg-create-branch branch 2 times, most recently from 08791a9 to b113839 Compare June 18, 2025 09:06
@wuwenchi
Copy link
Contributor Author

run buildall

@wuwenchi wuwenchi marked this pull request as ready for review June 18, 2025 09:06
@doris-robot
Copy link

TPC-H: Total hot run time: 33983 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b1138397f36ecfb74b1026688aeb591138ceccfb, data reload: false

------ Round 1 ----------------------------------
q1	17603	5119	4980	4980
q2	1931	298	184	184
q3	10299	1299	732	732
q4	10237	1057	518	518
q5	7515	2938	2327	2327
q6	175	167	136	136
q7	913	740	606	606
q8	9314	1285	1187	1187
q9	6815	5044	5126	5044
q10	6894	2397	1969	1969
q11	464	281	265	265
q12	343	352	209	209
q13	17791	3644	3068	3068
q14	230	238	208	208
q15	581	516	499	499
q16	429	430	386	386
q17	581	862	360	360
q18	7679	7268	7175	7175
q19	1217	963	569	569
q20	339	337	218	218
q21	4088	2591	2363	2363
q22	1028	1018	980	980
Total cold run time: 106466 ms
Total hot run time: 33983 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5063	5073	5068	5068
q2	242	337	224	224
q3	2191	2695	2314	2314
q4	1378	1765	1379	1379
q5	4182	4162	4473	4162
q6	220	170	126	126
q7	2047	1962	1768	1768
q8	2648	2540	2601	2540
q9	7115	7133	7152	7133
q10	3097	3287	2865	2865
q11	577	522	491	491
q12	683	768	623	623
q13	3479	3895	3273	3273
q14	281	295	265	265
q15	559	491	494	491
q16	450	486	436	436
q17	1147	1563	1374	1374
q18	7689	7419	7443	7419
q19	819	878	1061	878
q20	1974	2080	1890	1890
q21	4971	4524	4312	4312
q22	1077	1009	985	985
Total cold run time: 51889 ms
Total hot run time: 50016 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185994 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b1138397f36ecfb74b1026688aeb591138ceccfb, data reload: false

query1	998	392	387	387
query2	6542	1864	1834	1834
query3	6746	219	223	219
query4	26536	23650	23768	23650
query5	4365	646	467	467
query6	304	208	204	204
query7	4621	507	301	301
query8	266	225	217	217
query9	8630	2694	2683	2683
query10	480	350	276	276
query11	15628	14959	14968	14959
query12	157	111	108	108
query13	1665	532	434	434
query14	9140	6078	5977	5977
query15	225	198	176	176
query16	7309	609	480	480
query17	1197	733	589	589
query18	2020	419	318	318
query19	196	194	162	162
query20	176	118	120	118
query21	211	122	108	108
query22	4089	4147	4015	4015
query23	33945	33132	32920	32920
query24	8439	2389	2401	2389
query25	528	451	399	399
query26	1253	268	151	151
query27	2738	504	344	344
query28	4309	2173	2151	2151
query29	761	556	434	434
query30	288	222	189	189
query31	948	859	753	753
query32	75	62	64	62
query33	557	371	304	304
query34	794	835	540	540
query35	773	805	742	742
query36	991	982	883	883
query37	109	104	75	75
query38	4071	4182	4151	4151
query39	1450	1436	1424	1424
query40	208	133	106	106
query41	62	58	56	56
query42	121	132	109	109
query43	509	502	480	480
query44	1315	814	816	814
query45	177	178	180	178
query46	847	1026	630	630
query47	1762	1763	1718	1718
query48	394	427	315	315
query49	749	499	410	410
query50	655	670	402	402
query51	4186	4147	4103	4103
query52	107	107	100	100
query53	228	258	188	188
query54	576	578	510	510
query55	90	81	80	80
query56	310	285	293	285
query57	1169	1193	1110	1110
query58	271	255	242	242
query59	2569	2633	2496	2496
query60	327	316	307	307
query61	134	125	121	121
query62	797	746	654	654
query63	220	190	187	187
query64	4336	1007	671	671
query65	4248	4223	4213	4213
query66	1140	411	341	341
query67	15885	15577	15430	15430
query68	8415	898	547	547
query69	478	312	277	277
query70	1210	1066	1086	1066
query71	489	348	300	300
query72	5658	4670	5069	4670
query73	745	637	358	358
query74	9093	9069	8585	8585
query75	3868	3179	2729	2729
query76	3740	1199	764	764
query77	790	373	286	286
query78	10106	10145	9309	9309
query79	2473	844	606	606
query80	628	511	454	454
query81	480	264	235	235
query82	484	130	98	98
query83	283	258	234	234
query84	298	111	95	95
query85	781	377	313	313
query86	342	292	299	292
query87	4460	4462	4372	4372
query88	3002	2286	2347	2286
query89	402	324	285	285
query90	1940	222	268	222
query91	145	147	118	118
query92	75	63	58	58
query93	1189	988	595	595
query94	671	406	285	285
query95	365	291	295	291
query96	504	563	283	283
query97	2757	2766	2654	2654
query98	234	209	205	205
query99	1467	1402	1273	1273
Total cold run time: 274494 ms
Total hot run time: 185994 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.45 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit b1138397f36ecfb74b1026688aeb591138ceccfb, data reload: false

query1	0.03	0.04	0.03
query2	0.07	0.04	0.03
query3	0.24	0.06	0.07
query4	1.61	0.10	0.10
query5	0.42	0.43	0.41
query6	1.16	0.65	0.66
query7	0.02	0.02	0.02
query8	0.05	0.04	0.04
query9	0.59	0.52	0.51
query10	0.56	0.57	0.55
query11	0.16	0.10	0.11
query12	0.15	0.11	0.12
query13	0.63	0.62	0.61
query14	0.80	0.83	0.82
query15	0.89	0.86	0.90
query16	0.38	0.40	0.39
query17	1.10	1.06	1.04
query18	0.24	0.21	0.21
query19	2.02	1.82	1.82
query20	0.02	0.01	0.01
query21	15.41	0.87	0.55
query22	0.76	1.21	0.69
query23	14.90	1.41	0.65
query24	7.56	1.32	0.61
query25	0.50	0.17	0.20
query26	0.58	0.16	0.14
query27	0.07	0.06	0.05
query28	9.24	0.91	0.45
query29	12.61	3.92	3.27
query30	0.25	0.09	0.07
query31	2.83	0.59	0.40
query32	3.23	0.56	0.48
query33	3.02	3.15	3.12
query34	15.97	5.36	4.84
query35	4.79	4.87	4.77
query36	0.69	0.49	0.48
query37	0.09	0.07	0.07
query38	0.06	0.04	0.03
query39	0.03	0.03	0.03
query40	0.18	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.04	0.03	0.03
Total cold run time: 104.06 s
Total hot run time: 29.45 s

@wuwenchi
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 33961 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit edbd112f964cd9bb16397d2509f7ee6525eab475, data reload: false

------ Round 1 ----------------------------------
q1	17624	5162	5012	5012
q2	1937	303	195	195
q3	10402	1322	760	760
q4	10301	1028	541	541
q5	8337	2324	2348	2324
q6	180	163	133	133
q7	871	743	613	613
q8	9360	1270	1135	1135
q9	6775	5103	5076	5076
q10	6879	2377	1967	1967
q11	499	289	271	271
q12	343	341	212	212
q13	17801	3703	3082	3082
q14	227	225	207	207
q15	571	490	472	472
q16	444	436	365	365
q17	575	857	363	363
q18	7653	7108	7099	7099
q19	1487	957	564	564
q20	326	336	223	223
q21	3725	2571	2373	2373
q22	1068	1042	974	974
Total cold run time: 107385 ms
Total hot run time: 33961 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5120	5075	5071	5071
q2	247	325	234	234
q3	2165	2672	2301	2301
q4	1358	1829	1397	1397
q5	4245	4113	4450	4113
q6	207	169	129	129
q7	2063	1956	1790	1790
q8	2586	2594	2463	2463
q9	7144	7078	7095	7078
q10	3147	3345	2862	2862
q11	566	512	489	489
q12	653	763	577	577
q13	3459	3858	3324	3324
q14	274	309	262	262
q15	513	491	486	486
q16	464	488	432	432
q17	1167	1547	1349	1349
q18	7721	7591	7477	7477
q19	805	826	899	826
q20	1980	2088	1930	1930
q21	4929	4582	4407	4407
q22	1077	1015	992	992
Total cold run time: 51890 ms
Total hot run time: 49989 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185428 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit edbd112f964cd9bb16397d2509f7ee6525eab475, data reload: false

query1	1050	395	400	395
query2	6524	1911	1896	1896
query3	6740	219	217	217
query4	26456	23281	23307	23281
query5	4295	607	477	477
query6	335	204	204	204
query7	4633	493	293	293
query8	270	236	219	219
query9	8628	2637	2635	2635
query10	464	348	300	300
query11	15660	15037	14821	14821
query12	167	112	100	100
query13	1653	531	449	449
query14	8729	6141	6229	6141
query15	207	189	180	180
query16	7364	629	476	476
query17	1213	710	575	575
query18	1989	438	300	300
query19	182	181	154	154
query20	123	115	114	114
query21	213	123	105	105
query22	4114	4229	4049	4049
query23	34061	32922	33016	32922
query24	8434	2356	2370	2356
query25	534	446	392	392
query26	1228	266	152	152
query27	2777	490	342	342
query28	4331	2132	2111	2111
query29	761	549	430	430
query30	285	223	191	191
query31	946	903	750	750
query32	71	66	65	65
query33	569	378	303	303
query34	783	838	522	522
query35	770	813	749	749
query36	947	973	878	878
query37	112	105	78	78
query38	4065	4146	4053	4053
query39	1486	1400	1445	1400
query40	214	120	110	110
query41	65	61	58	58
query42	124	105	108	105
query43	507	538	484	484
query44	1290	809	804	804
query45	173	176	164	164
query46	823	1002	620	620
query47	1767	1813	1738	1738
query48	382	420	311	311
query49	731	474	398	398
query50	643	662	397	397
query51	4184	4130	4123	4123
query52	111	105	97	97
query53	223	253	186	186
query54	574	574	499	499
query55	86	82	83	82
query56	311	294	282	282
query57	1187	1196	1106	1106
query58	265	257	258	257
query59	2692	2789	2632	2632
query60	333	339	298	298
query61	124	125	130	125
query62	800	730	658	658
query63	227	194	191	191
query64	4490	1009	657	657
query65	4224	4174	4148	4148
query66	1138	400	309	309
query67	15833	15425	15384	15384
query68	8073	887	519	519
query69	463	304	266	266
query70	1214	1208	1089	1089
query71	453	324	309	309
query72	5577	4663	4633	4633
query73	688	562	348	348
query74	9190	9263	8825	8825
query75	3851	3246	2703	2703
query76	3686	1182	749	749
query77	800	386	294	294
query78	10028	10139	9218	9218
query79	2500	814	642	642
query80	690	499	451	451
query81	495	251	224	224
query82	445	128	99	99
query83	286	260	235	235
query84	295	116	91	91
query85	807	344	313	313
query86	386	308	271	271
query87	4365	4402	4262	4262
query88	3432	2309	2284	2284
query89	389	310	292	292
query90	1830	211	211	211
query91	145	141	112	112
query92	74	62	64	62
query93	1912	962	586	586
query94	661	410	308	308
query95	380	291	285	285
query96	492	561	281	281
query97	2710	2766	2689	2689
query98	238	213	203	203
query99	1414	1403	1276	1276
Total cold run time: 274659 ms
Total hot run time: 185428 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.13 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit edbd112f964cd9bb16397d2509f7ee6525eab475, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.23	0.08	0.06
query4	1.61	0.10	0.10
query5	0.42	0.41	0.41
query6	1.16	0.66	0.67
query7	0.03	0.01	0.02
query8	0.04	0.04	0.04
query9	0.57	0.53	0.51
query10	0.57	0.57	0.58
query11	0.16	0.11	0.12
query12	0.15	0.11	0.12
query13	0.62	0.61	0.61
query14	0.82	0.81	0.82
query15	0.90	0.87	0.90
query16	0.39	0.38	0.38
query17	1.05	1.12	1.08
query18	0.23	0.21	0.21
query19	1.99	1.87	1.88
query20	0.02	0.01	0.02
query21	15.42	0.90	0.55
query22	0.75	1.22	0.62
query23	14.98	1.29	0.63
query24	7.11	2.12	0.39
query25	0.38	0.17	0.09
query26	0.57	0.17	0.14
query27	0.07	0.06	0.06
query28	9.75	0.85	0.45
query29	12.56	4.01	3.28
query30	0.25	0.09	0.07
query31	2.82	0.60	0.39
query32	3.23	0.55	0.48
query33	3.13	3.22	3.06
query34	16.05	5.48	4.78
query35	4.84	4.83	4.84
query36	0.68	0.50	0.49
query37	0.10	0.07	0.07
query38	0.06	0.04	0.04
query39	0.04	0.03	0.02
query40	0.18	0.14	0.14
query41	0.08	0.02	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.17 s
Total hot run time: 29.13 s

@morningman morningman force-pushed the opt-iceberg-create-branch branch from edbd112 to e1eb917 Compare June 24, 2025 05:04
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34614 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a5f1a1a89c2c1c10586cf077fd19361a14e00943, data reload: false

------ Round 1 ----------------------------------
q1	17661	5249	5172	5172
q2	1946	324	187	187
q3	10545	1366	731	731
q4	10411	1004	522	522
q5	9054	2433	2393	2393
q6	206	164	132	132
q7	926	747	610	610
q8	9316	1297	1115	1115
q9	6916	5116	5134	5116
q10	6873	2393	1972	1972
q11	507	296	284	284
q12	354	348	218	218
q13	17766	3640	3124	3124
q14	230	231	215	215
q15	544	490	480	480
q16	442	427	378	378
q17	615	871	433	433
q18	7744	7313	7228	7228
q19	1220	963	565	565
q20	351	347	230	230
q21	4169	3257	2536	2536
q22	1091	1033	973	973
Total cold run time: 108887 ms
Total hot run time: 34614 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5131	5117	5147	5117
q2	251	324	229	229
q3	2197	2675	2300	2300
q4	1353	1778	1372	1372
q5	4236	4374	4425	4374
q6	213	173	128	128
q7	1986	1896	1738	1738
q8	2648	2561	2520	2520
q9	7227	7239	7206	7206
q10	3062	3295	2826	2826
q11	584	516	506	506
q12	662	741	588	588
q13	3528	3878	3254	3254
q14	296	297	277	277
q15	513	482	491	482
q16	455	485	437	437
q17	1147	1503	1392	1392
q18	7713	7476	7539	7476
q19	808	797	856	797
q20	1964	2030	1887	1887
q21	4936	4387	4341	4341
q22	1073	1042	995	995
Total cold run time: 51983 ms
Total hot run time: 50242 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 186466 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a5f1a1a89c2c1c10586cf077fd19361a14e00943, data reload: false

query1	1000	413	413	413
query2	6553	1859	1852	1852
query3	6745	226	222	222
query4	26011	23790	23301	23301
query5	4374	629	492	492
query6	306	211	191	191
query7	4634	515	300	300
query8	285	231	217	217
query9	8638	2656	2664	2656
query10	473	353	307	307
query11	15447	15129	14763	14763
query12	158	114	110	110
query13	1668	550	428	428
query14	8772	6130	6197	6130
query15	209	192	186	186
query16	7288	624	464	464
query17	1216	749	598	598
query18	1989	425	314	314
query19	198	203	167	167
query20	122	123	117	117
query21	219	129	116	116
query22	4128	4061	4166	4061
query23	34088	33142	33057	33057
query24	8409	2399	2392	2392
query25	545	474	406	406
query26	1234	273	154	154
query27	2750	509	355	355
query28	4345	2158	2138	2138
query29	786	561	445	445
query30	287	219	193	193
query31	928	855	753	753
query32	74	71	73	71
query33	572	356	301	301
query34	812	853	541	541
query35	800	841	737	737
query36	967	969	886	886
query37	111	102	77	77
query38	4195	4154	4054	4054
query39	1505	1445	1415	1415
query40	212	119	108	108
query41	62	60	58	58
query42	128	112	120	112
query43	485	491	463	463
query44	1345	857	846	846
query45	180	170	169	169
query46	847	1026	628	628
query47	1769	1798	1727	1727
query48	385	415	317	317
query49	747	499	404	404
query50	654	686	418	418
query51	4102	4183	4184	4183
query52	115	108	106	106
query53	232	262	187	187
query54	597	582	519	519
query55	95	86	91	86
query56	304	302	294	294
query57	1162	1201	1132	1132
query58	269	261	276	261
query59	2612	2674	2510	2510
query60	334	321	316	316
query61	130	155	128	128
query62	787	753	648	648
query63	228	188	189	188
query64	4350	1035	717	717
query65	4286	4179	4196	4179
query66	1175	460	328	328
query67	15757	15662	15366	15366
query68	8543	901	545	545
query69	470	316	278	278
query70	1173	1130	1118	1118
query71	461	341	302	302
query72	5548	4791	4725	4725
query73	700	605	353	353
query74	9146	9186	8912	8912
query75	3958	3216	2775	2775
query76	3709	1212	763	763
query77	793	390	301	301
query78	10154	10288	9266	9266
query79	2271	856	592	592
query80	625	526	478	478
query81	483	275	234	234
query82	418	127	100	100
query83	289	260	236	236
query84	300	113	111	111
query85	863	362	318	318
query86	357	263	268	263
query87	4430	4486	4387	4387
query88	3144	2343	2301	2301
query89	380	319	300	300
query90	1925	225	219	219
query91	142	143	118	118
query92	76	62	58	58
query93	1120	985	609	609
query94	675	396	319	319
query95	379	302	288	288
query96	504	566	284	284
query97	2707	2759	2648	2648
query98	235	211	199	199
query99	1421	1391	1288	1288
Total cold run time: 273589 ms
Total hot run time: 186466 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 29.77 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit a5f1a1a89c2c1c10586cf077fd19361a14e00943, data reload: false

query1	0.04	0.03	0.03
query2	0.08	0.04	0.03
query3	0.24	0.07	0.06
query4	1.61	0.11	0.10
query5	0.44	0.44	0.43
query6	1.17	0.67	0.67
query7	0.02	0.02	0.02
query8	0.04	0.04	0.03
query9	0.57	0.52	0.54
query10	0.58	0.58	0.56
query11	0.16	0.12	0.11
query12	0.15	0.11	0.12
query13	0.64	0.61	0.61
query14	0.80	0.82	0.84
query15	0.92	0.86	0.88
query16	0.41	0.38	0.39
query17	1.04	1.07	1.09
query18	0.23	0.21	0.22
query19	1.98	1.86	1.82
query20	0.01	0.01	0.02
query21	15.39	0.90	0.54
query22	0.76	1.28	0.76
query23	14.73	1.43	0.62
query24	7.03	1.71	0.91
query25	0.49	0.20	0.08
query26	0.51	0.18	0.14
query27	0.08	0.05	0.05
query28	10.20	0.90	0.45
query29	12.59	4.03	3.39
query30	0.25	0.09	0.06
query31	2.83	0.60	0.39
query32	3.23	0.54	0.46
query33	3.15	3.05	3.09
query34	16.08	5.45	4.79
query35	4.81	4.80	4.82
query36	0.71	0.51	0.50
query37	0.09	0.07	0.08
query38	0.05	0.04	0.04
query39	0.03	0.02	0.03
query40	0.19	0.14	0.13
query41	0.08	0.03	0.02
query42	0.03	0.03	0.02
query43	0.03	0.03	0.03
Total cold run time: 104.47 s
Total hot run time: 29.77 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 42.42% (140/330) 🎉
Increment coverage report
Complete coverage report

wuwenchi and others added 5 commits June 24, 2025 17:29
@wuwenchi @morningman
fix
5a91659 
fix

fix

fix style

test

fix

refactor interface
@morningman
1
a9b3598
@morningman
add test
33ad233
@morningman
java odc
0b6e963
@morningman
java odc
0ebe92d
@morningman morningman force-pushed the opt-iceberg-create-branch branch from a5f1a1a to 02d730c Compare June 24, 2025 09:37
@morningman
Copy link
Contributor

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 34097 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 153806a502cd30b3f69b8a1e9619f45acf6f6ff9, data reload: false

------ Round 1 ----------------------------------
q1	17673	5173	5082	5082
q2	1928	296	186	186
q3	10614	1306	713	713
q4	10327	995	532	532
q5	8746	2382	2385	2382
q6	191	162	130	130
q7	891	752	600	600
q8	9293	1309	1146	1146
q9	6851	5099	5228	5099
q10	6882	2386	1978	1978
q11	501	300	275	275
q12	349	346	217	217
q13	17777	3668	3062	3062
q14	227	227	220	220
q15	563	480	480	480
q16	425	431	371	371
q17	598	869	365	365
q18	7752	7194	7268	7194
q19	1290	952	539	539
q20	336	335	224	224
q21	4057	3199	2349	2349
q22	1087	1013	953	953
Total cold run time: 108358 ms
Total hot run time: 34097 ms

----- Round 2, with runtime_filter_mode=off -----
q1	5097	5061	5258	5061
q2	245	316	224	224
q3	2156	2615	2322	2322
q4	1370	1780	1381	1381
q5	4364	4394	4367	4367
q6	213	169	126	126
q7	1986	1895	1772	1772
q8	2570	2581	2499	2499
q9	7115	7137	6965	6965
q10	3023	3308	2822	2822
q11	574	517	511	511
q12	667	746	636	636
q13	3519	3865	3325	3325
q14	293	295	279	279
q15	522	474	483	474
q16	453	494	442	442
q17	1139	1595	1331	1331
q18	7344	7387	7166	7166
q19	798	769	878	769
q20	1935	1947	1824	1824
q21	4656	4375	4249	4249
q22	1076	1023	972	972
Total cold run time: 51115 ms
Total hot run time: 49517 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 185314 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 153806a502cd30b3f69b8a1e9619f45acf6f6ff9, data reload: false

query1	993	389	406	389
query2	6559	1808	1813	1808
query3	6765	217	218	217
query4	26404	23841	22843	22843
query5	4351	619	458	458
query6	330	195	183	183
query7	4626	495	283	283
query8	250	218	207	207
query9	8616	2602	2601	2601
query10	489	337	258	258
query11	15225	15076	14806	14806
query12	160	111	108	108
query13	1662	538	425	425
query14	8999	5994	6112	5994
query15	215	188	168	168
query16	7414	616	477	477
query17	1218	733	586	586
query18	2012	422	312	312
query19	201	195	165	165
query20	119	118	115	115
query21	218	131	110	110
query22	4319	4378	4298	4298
query23	33911	32945	33004	32945
query24	8524	2398	2377	2377
query25	541	454	387	387
query26	1245	264	148	148
query27	2761	504	343	343
query28	4298	2120	2099	2099
query29	753	554	419	419
query30	287	229	241	229
query31	899	817	780	780
query32	73	65	62	62
query33	551	358	334	334
query34	787	834	533	533
query35	782	855	729	729
query36	937	970	870	870
query37	108	100	78	78
query38	4095	4126	3967	3967
query39	1496	1405	1426	1405
query40	218	119	104	104
query41	64	61	60	60
query42	127	108	111	108
query43	470	475	450	450
query44	1307	821	821	821
query45	180	182	168	168
query46	856	1038	627	627
query47	1780	1825	1765	1765
query48	378	428	300	300
query49	748	469	384	384
query50	650	694	410	410
query51	4070	4117	4134	4117
query52	111	112	99	99
query53	220	257	184	184
query54	574	566	502	502
query55	86	83	83	83
query56	288	308	293	293
query57	1200	1196	1120	1120
query58	264	250	248	248
query59	2556	2740	2616	2616
query60	320	318	314	314
query61	127	122	125	122
query62	793	713	660	660
query63	214	186	184	184
query64	4338	989	676	676
query65	4282	4167	4146	4146
query66	1144	410	320	320
query67	15983	15712	15582	15582
query68	7969	888	528	528
query69	469	318	265	265
query70	1243	1131	1084	1084
query71	458	321	309	309
query72	5594	4711	4740	4711
query73	673	583	349	349
query74	8903	8826	8975	8826
query75	3816	3167	2665	2665
query76	3712	1179	752	752
query77	789	384	294	294
query78	10167	10288	9426	9426
query79	2179	807	593	593
query80	574	508	455	455
query81	504	268	282	268
query82	467	137	100	100
query83	251	246	238	238
query84	256	107	89	89
query85	780	354	314	314
query86	384	306	288	288
query87	4418	4457	4254	4254
query88	3725	2270	2255	2255
query89	390	314	276	276
query90	1838	206	204	204
query91	141	141	114	114
query92	79	56	57	56
query93	1667	956	590	590
query94	664	389	301	301
query95	376	288	281	281
query96	488	568	276	276
query97	2722	2754	2698	2698
query98	238	212	194	194
query99	1400	1425	1282	1282
Total cold run time: 273875 ms
Total hot run time: 185314 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.01 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 153806a502cd30b3f69b8a1e9619f45acf6f6ff9, data reload: false

query1	0.03	0.03	0.02
query2	0.11	0.05	0.04
query3	0.28	0.05	0.06
query4	1.60	0.07	0.08
query5	0.43	0.41	0.41
query6	1.17	0.64	0.66
query7	0.02	0.02	0.02
query8	0.06	0.04	0.05
query9	0.63	0.53	0.52
query10	0.57	0.58	0.57
query11	0.25	0.13	0.13
query12	0.25	0.16	0.14
query13	0.64	0.63	0.63
query14	0.80	0.83	0.82
query15	0.97	0.87	0.89
query16	0.38	0.39	0.38
query17	1.07	1.07	1.06
query18	0.23	0.22	0.23
query19	2.01	1.86	1.92
query20	0.01	0.02	0.01
query21	15.35	0.96	0.68
query22	0.92	1.05	0.82
query23	14.71	1.50	0.76
query24	5.28	0.57	0.30
query25	0.15	0.10	0.08
query26	0.55	0.23	0.18
query27	0.09	0.09	0.09
query28	11.01	1.12	0.57
query29	12.56	4.10	3.38
query30	0.28	0.09	0.07
query31	2.83	0.64	0.43
query32	3.26	0.60	0.50
query33	3.13	3.06	3.12
query34	16.55	5.39	4.67
query35	4.80	4.77	4.76
query36	0.64	0.51	0.50
query37	0.20	0.18	0.17
query38	0.17	0.16	0.15
query39	0.04	0.04	0.05
query40	0.19	0.17	0.16
query41	0.11	0.05	0.05
query42	0.06	0.05	0.06
query43	0.05	0.05	0.04
Total cold run time: 104.44 s
Total hot run time: 30.01 s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jun 24, 2025
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman morningman merged commit 91d7da6 into apache:master Jun 24, 2025
27 of 29 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@CalvinKirs CalvinKirs CalvinKirs approved these changes

@suxiaogang223 suxiaogang223 suxiaogang223 approved these changes

@hubgeter hubgeter hubgeter approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. dev/3.1.x dev/3.1.x-conflict reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
8 participants
@wuwenchi @Thearas @doris-robot @morningman @hello-stephen @CalvinKirs @suxiaogang223 @hubgeter