Skip to content

[opt](multi-catalog) Refactor data lake reader.#61485

Draft
kaka11chen wants to merge 1 commit intoapache:masterfrom
kaka11chen:iceberg_update_delete_merge_into_poc
Draft

[opt](multi-catalog) Refactor data lake reader.#61485
kaka11chen wants to merge 1 commit intoapache:masterfrom
kaka11chen:iceberg_update_delete_merge_into_poc

Conversation

@kaka11chen
Copy link
Contributor

@kaka11chen kaka11chen commented Mar 18, 2026

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason

  • Behavior changed:

    • No.
    • Yes.

  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 18, 2026

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 18, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 18, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.37% (32268/50130)
Region Coverage 65.25% (16155/24760)
Branch Coverage 55.67% (8605/15456)

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 18, 2026

FE UT Coverage Report

Increment line coverage 42.76% (809/1892) 🎉
Increment coverage report
Complete coverage report

@doris-robot
Copy link

doris-robot commented Mar 18, 2026

TPC-H: Total hot run time: 27028 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 65fb6e525ff302b594321d888768971e70245287, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17628	4465	4297	4297
q2	q3	10638	819	531	531
q4	4691	355	259	259
q5	7600	1260	1059	1059
q6	181	177	148	148
q7	807	892	716	716
q8	9329	1499	1395	1395
q9	4891	4817	4716	4716
q10	6263	1941	1679	1679
q11	492	270	242	242
q12	716	601	469	469
q13	18047	2934	2156	2156
q14	236	235	212	212
q15	q16	750	726	668	668
q17	754	830	479	479
q18	5982	5506	5207	5207
q19	1111	997	625	625
q20	549	498	384	384
q21	4321	1826	1458	1458
q22	546	364	328	328
Total cold run time: 95532 ms
Total hot run time: 27028 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4761	4585	4589	4585
q2	q3	3957	4348	3828	3828
q4	872	1195	764	764
q5	4068	4428	4347	4347
q6	187	185	143	143
q7	1804	1678	1513	1513
q8	2514	2749	2571	2571
q9	7858	7572	7440	7440
q10	3828	3990	3591	3591
q11	513	434	410	410
q12	504	583	456	456
q13	2731	3204	2290	2290
q14	278	295	268	268
q15	q16	693	750	722	722
q17	1441	1355	1386	1355
q18	7097	6963	6663	6663
q19	906	948	926	926
q20	2076	2162	2034	2034
q21	4003	3654	3318	3318
q22	449	411	379	379
Total cold run time: 50540 ms
Total hot run time: 47603 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2026

TPC-DS: Total hot run time: 169012 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 65fb6e525ff302b594321d888768971e70245287, data reload: false

query5	4335	653	509	509
query6	333	227	200	200
query7	4219	473	264	264
query8	349	254	232	232
query9	8694	2778	2747	2747
query10	508	403	342	342
query11	6959	5112	4908	4908
query12	183	129	140	129
query13	1266	463	343	343
query14	5793	3736	3516	3516
query14_1	2911	2767	2825	2767
query15	204	196	175	175
query16	1004	492	477	477
query17	1079	698	604	604
query18	2450	443	346	346
query19	217	208	187	187
query20	138	130	125	125
query21	218	132	110	110
query22	13416	14208	14691	14208
query23	16217	15866	15616	15616
query23_1	15577	15723	15355	15355
query24	7133	1630	1244	1244
query24_1	1237	1227	1217	1217
query25	564	460	415	415
query26	1248	261	149	149
query27	2790	483	294	294
query28	4493	1851	1844	1844
query29	864	574	478	478
query30	304	226	188	188
query31	1008	927	879	879
query32	83	74	70	70
query33	518	350	289	289
query34	892	930	533	533
query35	672	706	626	626
query36	1087	1114	955	955
query37	144	100	85	85
query38	3001	2925	2899	2899
query39	850	848	810	810
query39_1	800	788	786	786
query40	250	168	144	144
query41	66	65	64	64
query42	268	261	262	261
query43	249	254	220	220
query44	
query45	242	191	182	182
query46	877	982	613	613
query47	2599	2125	2770	2125
query48	323	313	232	232
query49	640	460	378	378
query50	705	274	217	217
query51	4076	4039	4047	4039
query52	262	276	256	256
query53	289	350	298	298
query54	308	277	279	277
query55	96	89	84	84
query56	324	326	324	324
query57	1908	1931	1669	1669
query58	283	277	271	271
query59	2825	2952	2757	2757
query60	349	348	332	332
query61	156	159	158	158
query62	634	594	544	544
query63	310	282	275	275
query64	5079	1325	1010	1010
query65	
query66	1466	455	364	364
query67	24537	24310	24185	24185
query68	
query69	407	316	300	300
query70	980	989	951	951
query71	343	310	317	310
query72	2753	2693	2600	2600
query73	541	548	321	321
query74	9549	9581	9413	9413
query75	2880	2796	2481	2481
query76	2311	1036	692	692
query77	383	419	325	325
query78	10907	11001	10475	10475
query79	1125	791	581	581
query80	876	671	580	580
query81	524	277	223	223
query82	1350	152	122	122
query83	390	277	253	253
query84	259	122	108	108
query85	990	586	541	541
query86	390	315	303	303
query87	3121	3203	3033	3033
query88	3540	2700	2678	2678
query89	441	380	356	356
query90	1837	181	179	179
query91	188	184	198	184
query92	77	76	74	74
query93	917	826	491	491
query94	515	327	307	307
query95	584	341	311	311
query96	643	515	226	226
query97	2470	2478	2439	2439
query98	230	225	218	218
query99	1004	969	891	891
Total cold run time: 249955 ms
Total hot run time: 169012 ms

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 18, 2026

run buildall

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from 65fb6e5 to c50ae32 Compare March 18, 2026 18:11
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 18, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Mar 18, 2026

TPC-H: Total hot run time: 27241 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit c50ae32b06758780df99a3a723e03ea94c9b19d6, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17629	4460	4267	4267
q2	q3	10623	810	539	539
q4	4681	369	252	252
q5	7586	1206	1022	1022
q6	179	181	149	149
q7	789	835	695	695
q8	9479	1472	1313	1313
q9	5659	4771	4731	4731
q10	6294	1956	1684	1684
q11	479	262	257	257
q12	710	582	471	471
q13	18062	2939	2207	2207
q14	227	229	210	210
q15	q16	745	741	661	661
q17	745	849	446	446
q18	6055	5442	5334	5334
q19	1330	969	631	631
q20	548	491	377	377
q21	4681	1890	1724	1724
q22	416	315	271	271
Total cold run time: 96917 ms
Total hot run time: 27241 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4623	4572	4541	4541
q2	q3	3994	4446	3841	3841
q4	884	1215	801	801
q5	4066	4383	4353	4353
q6	188	181	155	155
q7	1794	1661	1568	1568
q8	2505	2734	2645	2645
q9	7592	7459	7450	7450
q10	3773	4017	3560	3560
q11	517	430	431	430
q12	503	593	452	452
q13	2911	3170	2336	2336
q14	298	320	303	303
q15	q16	716	776	744	744
q17	1271	1356	1390	1356
q18	7297	6962	6788	6788
q19	882	904	909	904
q20	2057	2179	2000	2000
q21	3987	3555	3331	3331
q22	497	446	384	384
Total cold run time: 50355 ms
Total hot run time: 47942 ms

@doris-robot
Copy link

doris-robot commented Mar 18, 2026

TPC-DS: Total hot run time: 169009 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit c50ae32b06758780df99a3a723e03ea94c9b19d6, data reload: false

query5	4332	650	515	515
query6	345	235	223	223
query7	4225	472	272	272
query8	354	243	230	230
query9	8739	2765	2762	2762
query10	491	415	346	346
query11	7023	5140	4879	4879
query12	188	136	130	130
query13	1278	462	344	344
query14	5769	3782	3503	3503
query14_1	2852	2850	2813	2813
query15	211	195	176	176
query16	983	486	468	468
query17	907	731	632	632
query18	2455	450	360	360
query19	222	223	188	188
query20	143	131	129	129
query21	213	135	111	111
query22	13267	14044	14447	14044
query23	16124	15847	15630	15630
query23_1	16055	15911	15826	15826
query24	7311	1610	1222	1222
query24_1	1239	1247	1258	1247
query25	616	468	417	417
query26	1253	266	152	152
query27	2780	478	299	299
query28	4485	1844	1840	1840
query29	847	567	485	485
query30	300	229	187	187
query31	1024	925	869	869
query32	81	76	68	68
query33	522	344	294	294
query34	927	859	532	532
query35	647	671	599	599
query36	1021	1102	972	972
query37	137	96	85	85
query38	2921	2969	2917	2917
query39	856	840	826	826
query39_1	802	793	797	793
query40	236	157	137	137
query41	64	59	59	59
query42	261	255	260	255
query43	246	252	233	233
query44	
query45	191	193	181	181
query46	876	982	620	620
query47	2995	2176	2063	2063
query48	328	313	246	246
query49	642	473	386	386
query50	695	273	219	219
query51	4096	3983	4038	3983
query52	263	268	257	257
query53	290	337	285	285
query54	302	273	263	263
query55	101	96	85	85
query56	323	327	315	315
query57	1902	1628	1791	1628
query58	290	275	277	275
query59	2762	2931	2776	2776
query60	369	346	332	332
query61	160	155	156	155
query62	639	593	538	538
query63	306	287	283	283
query64	5109	1279	1011	1011
query65	
query66	1475	451	357	357
query67	24487	24326	24385	24326
query68	
query69	414	316	299	299
query70	964	949	924	924
query71	356	325	305	305
query72	2905	2714	2555	2555
query73	544	544	325	325
query74	9643	9578	9372	9372
query75	2860	2772	2470	2470
query76	2291	1038	689	689
query77	361	382	305	305
query78	11003	11136	10455	10455
query79	1100	748	580	580
query80	1326	612	545	545
query81	552	259	219	219
query82	1213	150	122	122
query83	343	282	244	244
query84	302	113	95	95
query85	936	532	447	447
query86	422	347	291	291
query87	3165	3105	3007	3007
query88	3567	2664	2645	2645
query89	422	368	343	343
query90	1996	181	186	181
query91	175	164	141	141
query92	76	73	69	69
query93	896	859	501	501
query94	642	321	304	304
query95	594	344	340	340
query96	645	513	228	228
query97	2456	2490	2413	2413
query98	241	226	219	219
query99	1028	1018	895	895
Total cold run time: 250813 ms
Total hot run time: 169009 ms

@kaka11chen kaka11chen changed the title [feature](iceberg) Implements iceberg update & delete & merge into. [opt](multi-catalog) Refactor data lake reader. Mar 20, 2026
@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch 2 times, most recently from c4e1eda to 52b66bd Compare March 21, 2026 05:25
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 21, 2026

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 21, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.39% (32278/50130)
Region Coverage 65.29% (16165/24760)
Branch Coverage 55.68% (8606/15456)

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from 52b66bd to 32c2d9d Compare March 21, 2026 11:46
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 21, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Mar 21, 2026

TPC-H: Total hot run time: 26546 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 32c2d9d4083d94d6af34b984f6e62b0fa16a9d82, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17186	4474	4285	4285
q2	q3	10416	794	509	509
q4	4663	359	249	249
q5	7579	1212	1041	1041
q6	174	170	147	147
q7	777	849	683	683
q8	9347	1483	1349	1349
q9	4820	4648	4706	4648
q10	6241	1898	1666	1666
q11	469	255	248	248
q12	739	597	465	465
q13	18028	2916	2147	2147
q14	221	247	209	209
q15	q16	755	745	667	667
q17	772	850	458	458
q18	6014	5420	5166	5166
q19	1148	990	610	610
q20	554	492	369	369
q21	4374	1819	1390	1390
q22	354	288	240	240
Total cold run time: 94631 ms
Total hot run time: 26546 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5040	4693	4672	4672
q2	q3	3915	4330	3872	3872
q4	912	1186	790	790
q5	4102	4379	4362	4362
q6	181	174	135	135
q7	1788	1717	1560	1560
q8	2474	2669	2561	2561
q9	7779	7378	7339	7339
q10	3843	4076	3585	3585
q11	506	448	423	423
q12	484	587	436	436
q13	2724	3114	2455	2455
q14	298	318	289	289
q15	q16	760	766	706	706
q17	1146	1369	1345	1345
q18	7230	7028	6694	6694
q19	925	937	878	878
q20	2065	2180	1968	1968
q21	3933	3534	3318	3318
q22	479	439	372	372
Total cold run time: 50584 ms
Total hot run time: 47760 ms

@doris-robot
Copy link

doris-robot commented Mar 21, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.37% (32268/50130)
Region Coverage 65.27% (16162/24760)
Branch Coverage 55.67% (8604/15456)

@doris-robot
Copy link

doris-robot commented Mar 21, 2026

TPC-DS: Total hot run time: 167840 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 32c2d9d4083d94d6af34b984f6e62b0fa16a9d82, data reload: false

query5	4347	642	532	532
query6	340	235	227	227
query7	4217	460	262	262
query8	349	254	234	234
query9	8725	2713	2698	2698
query10	508	389	337	337
query11	7016	5132	4881	4881
query12	196	128	126	126
query13	1279	462	371	371
query14	5821	3629	3496	3496
query14_1	2868	2807	2820	2807
query15	206	193	178	178
query16	983	473	453	453
query17	913	720	615	615
query18	2451	448	360	360
query19	214	213	184	184
query20	141	129	129	129
query21	217	130	112	112
query22	13288	13353	13184	13184
query23	15843	15483	15608	15483
query23_1	15931	15934	15860	15860
query24	7563	1691	1298	1298
query24_1	1325	1293	1284	1284
query25	611	564	510	510
query26	1317	297	169	169
query27	3166	520	347	347
query28	4562	1941	1896	1896
query29	868	759	539	539
query30	319	247	207	207
query31	1077	1011	942	942
query32	86	73	71	71
query33	505	323	279	279
query34	908	865	538	538
query35	663	684	586	586
query36	1060	1143	983	983
query37	134	91	82	82
query38	2922	2932	2839	2839
query39	845	844	804	804
query39_1	788	785	791	785
query40	228	149	133	133
query41	63	57	59	57
query42	257	251	261	251
query43	237	240	218	218
query44	
query45	197	184	184	184
query46	869	999	614	614
query47	2120	2112	2086	2086
query48	311	313	223	223
query49	662	464	375	375
query50	676	273	205	205
query51	4039	4098	3950	3950
query52	263	262	255	255
query53	296	338	285	285
query54	309	279	273	273
query55	88	91	85	85
query56	324	325	299	299
query57	1923	1870	1683	1683
query58	287	272	272	272
query59	2786	2952	2752	2752
query60	347	336	317	317
query61	159	154	151	151
query62	621	600	538	538
query63	304	276	289	276
query64	4920	1280	970	970
query65	
query66	1456	448	361	361
query67	24208	24266	24148	24148
query68	
query69	414	303	285	285
query70	961	988	936	936
query71	335	299	286	286
query72	2847	2628	2424	2424
query73	530	533	324	324
query74	9624	9578	9402	9402
query75	2828	2735	2462	2462
query76	2184	1015	704	704
query77	356	372	313	313
query78	11010	11051	10435	10435
query79	3084	736	566	566
query80	1749	622	573	573
query81	577	260	222	222
query82	954	149	116	116
query83	340	263	246	246
query84	294	122	95	95
query85	908	494	434	434
query86	509	326	286	286
query87	3129	3149	3027	3027
query88	3547	2652	2640	2640
query89	440	364	342	342
query90	2096	183	175	175
query91	169	160	138	138
query92	88	77	74	74
query93	2341	812	505	505
query94	649	324	285	285
query95	585	410	320	320
query96	666	509	228	228
query97	2454	2477	2381	2381
query98	254	229	221	221
query99	933	998	922	922
Total cold run time: 253625 ms
Total hot run time: 167840 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 21, 2026

FE UT Coverage Report

Increment line coverage 38.69% (785/2029) 🎉
Increment coverage report
Complete coverage report

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from 32c2d9d to fd4b119 Compare March 21, 2026 16:29
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 21, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Mar 21, 2026

TPC-H: Total hot run time: 26672 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd4b119501a6aa325f3cae5b3b73b013667ba4e6, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	16981	4394	4278	4278
q2	q3	10394	759	520	520
q4	4702	358	249	249
q5	7602	1220	1029	1029
q6	177	173	144	144
q7	802	830	666	666
q8	9390	1474	1359	1359
q9	4776	4718	4677	4677
q10	6249	1903	1639	1639
q11	461	260	237	237
q12	745	599	482	482
q13	18054	2966	2175	2175
q14	228	229	213	213
q15	q16	734	739	675	675
q17	747	849	430	430
q18	6148	5481	5296	5296
q19	1119	963	625	625
q20	537	478	374	374
q21	4455	1818	1355	1355
q22	341	290	249	249
Total cold run time: 94642 ms
Total hot run time: 26672 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	5006	4693	4592	4592
q2	q3	3885	4340	3816	3816
q4	1000	1231	775	775
q5	4080	4426	4373	4373
q6	184	174	143	143
q7	1799	1647	1509	1509
q8	2529	2711	2542	2542
q9	7666	7321	7484	7321
q10	3739	3945	3567	3567
q11	512	434	421	421
q12	498	588	450	450
q13	2748	3126	2352	2352
q14	294	317	282	282
q15	q16	774	749	712	712
q17	1140	1334	1339	1334
q18	7289	6902	6576	6576
q19	914	884	912	884
q20	2073	2185	2030	2030
q21	4001	3437	3552	3437
q22	465	432	367	367
Total cold run time: 50596 ms
Total hot run time: 47483 ms

@doris-robot
Copy link

doris-robot commented Mar 21, 2026

TPC-DS: Total hot run time: 168017 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd4b119501a6aa325f3cae5b3b73b013667ba4e6, data reload: false

query5	4353	647	508	508
query6	356	253	211	211
query7	4233	469	271	271
query8	340	246	232	232
query9	8755	2727	2698	2698
query10	486	410	354	354
query11	7013	5064	4916	4916
query12	187	127	128	127
query13	1298	455	340	340
query14	5623	3745	3472	3472
query14_1	2868	2826	2812	2812
query15	203	194	173	173
query16	964	477	441	441
query17	902	729	634	634
query18	2454	474	355	355
query19	216	220	181	181
query20	134	127	129	127
query21	208	133	113	113
query22	13228	13958	14754	13958
query23	16271	15589	15621	15589
query23_1	15749	15492	15477	15477
query24	6998	1608	1230	1230
query24_1	1230	1226	1271	1226
query25	534	461	410	410
query26	1243	261	145	145
query27	2792	474	293	293
query28	4408	1837	1823	1823
query29	804	555	474	474
query30	295	236	189	189
query31	1012	948	871	871
query32	87	70	71	70
query33	510	328	288	288
query34	886	864	524	524
query35	623	681	581	581
query36	1071	1152	951	951
query37	130	92	78	78
query38	2946	2929	2922	2922
query39	853	831	820	820
query39_1	800	795	788	788
query40	230	155	139	139
query41	62	58	60	58
query42	263	252	258	252
query43	233	251	219	219
query44	
query45	237	192	185	185
query46	872	978	609	609
query47	2090	2133	2038	2038
query48	322	313	219	219
query49	633	452	381	381
query50	693	273	221	221
query51	4179	4030	4001	4001
query52	271	261	260	260
query53	288	334	282	282
query54	292	270	262	262
query55	94	88	89	88
query56	313	323	315	315
query57	1923	1730	1561	1561
query58	287	274	271	271
query59	2795	2979	2737	2737
query60	337	341	324	324
query61	155	150	152	150
query62	635	589	530	530
query63	315	275	279	275
query64	4917	1245	986	986
query65	
query66	1463	449	367	367
query67	24186	24262	24197	24197
query68	
query69	409	298	284	284
query70	956	961	966	961
query71	335	306	300	300
query72	2918	2731	2504	2504
query73	539	542	317	317
query74	9640	9536	9365	9365
query75	2851	2733	2476	2476
query76	2152	1021	671	671
query77	366	374	306	306
query78	10853	11158	10451	10451
query79	1107	774	589	589
query80	1060	623	545	545
query81	538	255	226	226
query82	1307	153	116	116
query83	343	254	248	248
query84	296	126	103	103
query85	882	497	454	454
query86	405	297	299	297
query87	3135	3113	3012	3012
query88	3522	2635	2635	2635
query89	430	375	357	357
query90	1951	177	177	177
query91	168	165	135	135
query92	78	72	73	72
query93	921	839	495	495
query94	512	318	303	303
query95	583	344	368	344
query96	653	508	230	230
query97	2507	2463	2430	2430
query98	243	223	221	221
query99	909	1007	926	926
Total cold run time: 248359 ms
Total hot run time: 168017 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 21, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.39% (32281/50130)
Region Coverage 65.29% (16167/24760)
Branch Coverage 55.73% (8613/15456)

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 21, 2026

FE UT Coverage Report

Increment line coverage 38.69% (785/2029) 🎉
Increment coverage report
Complete coverage report

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from fd4b119 to a57820e Compare March 22, 2026 15:06
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 22, 2026

run buildall

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 22, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.36% (32266/50130)
Region Coverage 65.28% (16163/24760)
Branch Coverage 55.69% (8608/15456)

@doris-robot
Copy link

doris-robot commented Mar 22, 2026

TPC-H: Total hot run time: 27137 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit a57820e7837cf4f59e7a02861e407ae1855ad8dd, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17635	4506	4466	4466
q2	q3	10718	773	522	522
q4	4742	358	248	248
q5	8018	1214	1009	1009
q6	221	175	149	149
q7	815	856	678	678
q8	10484	1509	1340	1340
q9	5962	4814	4685	4685
q10	6426	1951	1648	1648
q11	503	269	246	246
q12	765	582	504	504
q13	18068	2951	2168	2168
q14	235	228	216	216
q15	q16	751	745	675	675
q17	755	799	503	503
q18	5863	5360	5292	5292
q19	1212	995	627	627
q20	539	537	386	386
q21	4490	2196	1513	1513
q22	397	320	262	262
Total cold run time: 98599 ms
Total hot run time: 27137 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4770	4714	4622	4622
q2	q3	3918	4365	3846	3846
q4	875	1214	798	798
q5	4077	4359	4339	4339
q6	184	182	142	142
q7	1728	1663	1590	1590
q8	2522	2756	2607	2607
q9	7531	7323	7417	7323
q10	3840	3957	3609	3609
q11	505	439	424	424
q12	496	609	441	441
q13	2881	3261	2414	2414
q14	301	301	281	281
q15	q16	721	792	738	738
q17	1195	1451	1336	1336
q18	7319	6798	7032	6798
q19	921	929	932	929
q20	2066	2145	1962	1962
q21	3996	3578	3372	3372
q22	468	436	375	375
Total cold run time: 50314 ms
Total hot run time: 47946 ms

@doris-robot
Copy link

doris-robot commented Mar 22, 2026

TPC-DS: Total hot run time: 168348 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit a57820e7837cf4f59e7a02861e407ae1855ad8dd, data reload: false

query5	4331	616	508	508
query6	354	233	210	210
query7	4235	489	281	281
query8	348	238	233	233
query9	8721	2721	2682	2682
query10	511	414	348	348
query11	7034	5098	4877	4877
query12	185	134	128	128
query13	1292	487	363	363
query14	5719	3704	3478	3478
query14_1	2847	2848	2843	2843
query15	210	190	186	186
query16	1019	478	470	470
query17	921	726	641	641
query18	2456	452	352	352
query19	226	218	206	206
query20	134	126	129	126
query21	218	138	109	109
query22	13157	14027	14679	14027
query23	16231	16054	15596	15596
query23_1	16167	15724	15495	15495
query24	7211	1643	1228	1228
query24_1	1242	1231	1255	1231
query25	608	463	400	400
query26	1239	262	148	148
query27	2780	479	288	288
query28	4479	1824	1844	1824
query29	841	566	475	475
query30	297	223	192	192
query31	1013	957	862	862
query32	88	70	70	70
query33	516	345	283	283
query34	889	864	515	515
query35	662	677	602	602
query36	1060	1086	968	968
query37	134	102	81	81
query38	2937	2900	2903	2900
query39	857	846	815	815
query39_1	789	791	797	791
query40	234	150	137	137
query41	63	57	59	57
query42	256	252	256	252
query43	233	252	224	224
query44	
query45	199	190	221	190
query46	884	978	610	610
query47	2543	2502	2072	2072
query48	322	328	238	238
query49	634	463	372	372
query50	683	267	210	210
query51	4049	3987	4081	3987
query52	258	269	254	254
query53	289	361	293	293
query54	300	274	268	268
query55	88	85	86	85
query56	314	321	314	314
query57	1941	1744	1823	1744
query58	285	279	269	269
query59	2761	2936	2754	2754
query60	356	357	320	320
query61	153	147	152	147
query62	632	577	547	547
query63	308	279	277	277
query64	5117	1274	1025	1025
query65	
query66	1460	447	353	353
query67	24208	24257	24276	24257
query68	
query69	409	311	281	281
query70	986	966	1009	966
query71	336	314	291	291
query72	2813	2666	2505	2505
query73	542	550	319	319
query74	9617	9523	9393	9393
query75	2861	2735	2491	2491
query76	2287	1037	672	672
query77	360	380	317	317
query78	10894	11067	10464	10464
query79	3148	748	573	573
query80	1790	634	542	542
query81	584	256	222	222
query82	985	149	117	117
query83	334	260	246	246
query84	303	117	97	97
query85	913	494	465	465
query86	534	336	302	302
query87	3140	3120	2972	2972
query88	3589	2677	2687	2677
query89	429	376	345	345
query90	2227	181	170	170
query91	165	164	136	136
query92	89	77	69	69
query93	1936	820	494	494
query94	641	314	283	283
query95	588	339	317	317
query96	636	509	224	224
query97	2442	2486	2383	2383
query98	243	222	223	222
query99	1015	998	898	898
Total cold run time: 253258 ms
Total hot run time: 168348 ms

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from a57820e to adae5a2 Compare March 23, 2026 07:29
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 23, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Mar 23, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.40% (32285/50130)
Region Coverage 65.26% (16158/24760)
Branch Coverage 55.75% (8617/15456)

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 23, 2026

FE UT Coverage Report

Increment line coverage 38.69% (785/2029) 🎉
Increment coverage report
Complete coverage report

@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 23, 2026

run external

@doris-robot
Copy link

doris-robot commented Mar 23, 2026

TPC-H: Total hot run time: 27163 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit adae5a2c00291d605b1647839faf1eb0e71be6ef, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17621	4538	4436	4436
q2	q3	10728	798	516	516
q4	4734	379	255	255
q5	8169	1239	1011	1011
q6	238	178	143	143
q7	832	838	688	688
q8	10607	1496	1352	1352
q9	6817	4839	4790	4790
q10	6343	1936	1652	1652
q11	484	250	237	237
q12	721	590	463	463
q13	18110	2929	2188	2188
q14	230	239	215	215
q15	q16	747	764	682	682
q17	755	876	426	426
q18	5978	5434	5293	5293
q19	1121	990	623	623
q20	555	497	380	380
q21	4799	1999	1545	1545
q22	374	319	268	268
Total cold run time: 99963 ms
Total hot run time: 27163 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4686	4575	4781	4575
q2	q3	3938	4360	3860	3860
q4	871	1200	784	784
q5	4118	4443	4349	4349
q6	193	170	138	138
q7	1794	1642	1540	1540
q8	2531	2818	2664	2664
q9	7643	7315	7370	7315
q10	3779	3989	3699	3699
q11	532	455	435	435
q12	499	592	467	467
q13	3160	3209	2350	2350
q14	284	323	293	293
q15	q16	778	768	726	726
q17	1169	1331	1380	1331
q18	7230	6919	6637	6637
q19	1069	955	908	908
q20	2114	2205	2009	2009
q21	4038	3478	3315	3315
q22	480	452	387	387
Total cold run time: 50906 ms
Total hot run time: 47782 ms

@doris-robot
Copy link

doris-robot commented Mar 23, 2026

TPC-DS: Total hot run time: 168873 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit adae5a2c00291d605b1647839faf1eb0e71be6ef, data reload: false

query5	4342	655	525	525
query6	337	248	228	228
query7	4226	482	266	266
query8	372	252	240	240
query9	8699	2737	2734	2734
query10	496	400	344	344
query11	6981	5089	4876	4876
query12	192	136	130	130
query13	1279	479	359	359
query14	5726	3688	3478	3478
query14_1	2855	2850	2821	2821
query15	208	198	183	183
query16	998	462	429	429
query17	920	763	639	639
query18	2441	466	366	366
query19	229	210	191	191
query20	139	132	133	132
query21	219	131	112	112
query22	13191	13933	15074	13933
query23	16199	15895	15730	15730
query23_1	15873	15669	16099	15669
query24	7307	1628	1215	1215
query24_1	1230	1222	1223	1222
query25	535	466	401	401
query26	1244	264	148	148
query27	2780	483	291	291
query28	4482	1854	1815	1815
query29	852	577	486	486
query30	305	226	187	187
query31	1019	974	865	865
query32	83	81	72	72
query33	525	341	329	329
query34	902	889	523	523
query35	639	677	599	599
query36	1086	1138	992	992
query37	133	94	83	83
query38	2968	2929	2902	2902
query39	855	827	811	811
query39_1	804	800	805	800
query40	237	163	144	144
query41	63	60	60	60
query42	259	263	256	256
query43	255	247	230	230
query44	
query45	200	194	182	182
query46	901	977	601	601
query47	2133	2143	2064	2064
query48	312	312	221	221
query49	640	472	389	389
query50	686	283	213	213
query51	4142	4038	4014	4014
query52	262	267	257	257
query53	286	342	283	283
query54	305	277	281	277
query55	94	96	82	82
query56	325	325	325	325
query57	1928	1674	1839	1674
query58	292	285	284	284
query59	2781	2932	2760	2760
query60	346	359	338	338
query61	155	150	146	146
query62	619	560	541	541
query63	303	286	273	273
query64	5080	1260	981	981
query65	
query66	1468	456	356	356
query67	24252	24270	24177	24177
query68	
query69	425	317	298	298
query70	980	978	926	926
query71	349	329	312	312
query72	2794	2629	2519	2519
query73	534	537	315	315
query74	9636	9588	9426	9426
query75	2850	2792	2466	2466
query76	2280	1032	654	654
query77	356	387	307	307
query78	10999	11252	10487	10487
query79	1111	800	564	564
query80	974	652	585	585
query81	531	268	226	226
query82	1330	151	118	118
query83	337	261	242	242
query84	304	121	104	104
query85	983	579	527	527
query86	403	327	308	308
query87	3129	3110	3091	3091
query88	3546	2704	2649	2649
query89	424	369	346	346
query90	1871	183	190	183
query91	170	162	141	141
query92	81	79	71	71
query93	909	843	487	487
query94	516	319	300	300
query95	588	337	330	330
query96	653	526	229	229
query97	2440	2458	2397	2397
query98	236	230	223	223
query99	1011	986	936	936
Total cold run time: 248580 ms
Total hot run time: 168873 ms

@kaka11chen kaka11chen force-pushed the iceberg_update_delete_merge_into_poc branch from adae5a2 to 663e5c3 Compare March 23, 2026 12:11
@kaka11chen
Copy link
Contributor Author

kaka11chen commented Mar 23, 2026

run buildall

@doris-robot
Copy link

doris-robot commented Mar 23, 2026

TPC-H: Total hot run time: 26817 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 663e5c3a765550684edce491c3fe286c3fa83218, data reload: false

------ Round 1 ----------------------------------
orders	Doris	NULL	NULL	0	0	0	NULL	0	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	17612	4436	4324	4324
q2	q3	10632	830	526	526
q4	4681	354	257	257
q5	7570	1217	1005	1005
q6	174	174	147	147
q7	772	848	656	656
q8	9321	1494	1349	1349
q9	4824	4744	4731	4731
q10	6248	1909	1637	1637
q11	469	257	250	250
q12	706	583	461	461
q13	18043	2918	2163	2163
q14	227	228	218	218
q15	q16	717	736	663	663
q17	730	872	423	423
q18	5927	5380	5285	5285
q19	1111	997	604	604
q20	551	481	381	381
q21	4433	1842	1415	1415
q22	342	322	423	322
Total cold run time: 95090 ms
Total hot run time: 26817 ms

----- Round 2, with runtime_filter_mode=off -----
orders	Doris	NULL	NULL	150000000	42	6422171781	NULL	22778155	NULL	NULL	2023-12-26 18:27:23	2023-12-26 18:42:55	NULL	utf-8	NULL	NULL	
============================================
q1	4789	4675	4547	4547
q2	q3	3897	4327	3812	3812
q4	894	1204	789	789
q5	4050	4405	4386	4386
q6	184	176	148	148
q7	1792	1647	1517	1517
q8	2477	2724	2578	2578
q9	7658	7302	7475	7302
q10	3734	3999	3700	3700
q11	525	447	423	423
q12	521	674	455	455
q13	2655	3164	2270	2270
q14	277	294	274	274
q15	q16	731	792	767	767
q17	1191	1398	1319	1319
q18	7184	6755	6562	6562
q19	923	873	983	873
q20	2101	2142	2020	2020
q21	3960	3478	3382	3382
q22	548	419	388	388
Total cold run time: 50091 ms
Total hot run time: 47512 ms

@doris-robot
Copy link

doris-robot commented Mar 23, 2026

TPC-DS: Total hot run time: 168166 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 663e5c3a765550684edce491c3fe286c3fa83218, data reload: false

query5	4328	623	499	499
query6	334	245	210	210
query7	4223	474	268	268
query8	349	252	234	234
query9	8725	2696	2718	2696
query10	553	377	347	347
query11	6992	5075	4867	4867
query12	187	133	123	123
query13	1274	458	344	344
query14	5771	3694	3467	3467
query14_1	2845	2833	2839	2833
query15	214	194	175	175
query16	987	474	462	462
query17	969	743	664	664
query18	2450	452	357	357
query19	223	216	186	186
query20	129	132	127	127
query21	216	129	113	113
query22	13125	14095	14741	14095
query23	16172	16040	15630	15630
query23_1	15612	15436	15358	15358
query24	7158	1621	1211	1211
query24_1	1272	1234	1208	1208
query25	532	471	400	400
query26	1245	265	144	144
query27	2795	478	336	336
query28	4488	1835	1830	1830
query29	824	571	483	483
query30	300	223	190	190
query31	996	947	872	872
query32	80	67	67	67
query33	528	335	295	295
query34	910	873	518	518
query35	651	682	596	596
query36	1078	1152	988	988
query37	137	97	83	83
query38	2945	2937	2875	2875
query39	861	854	810	810
query39_1	797	795	776	776
query40	226	155	137	137
query41	61	60	56	56
query42	257	255	251	251
query43	243	255	212	212
query44	
query45	196	190	183	183
query46	874	985	599	599
query47	2141	2165	2070	2070
query48	315	314	220	220
query49	622	487	382	382
query50	685	278	212	212
query51	4215	4046	4034	4034
query52	259	268	251	251
query53	290	338	282	282
query54	299	276	275	275
query55	95	88	81	81
query56	316	321	299	299
query57	1950	1904	1753	1753
query58	289	274	270	270
query59	2791	2987	2711	2711
query60	336	340	323	323
query61	151	153	156	153
query62	624	601	542	542
query63	312	279	276	276
query64	4984	1281	1005	1005
query65	
query66	1467	451	361	361
query67	24130	24236	24100	24100
query68	
query69	397	308	284	284
query70	888	963	969	963
query71	339	318	305	305
query72	2927	2876	2640	2640
query73	541	553	318	318
query74	9606	9535	9357	9357
query75	2867	2765	2491	2491
query76	2273	1025	696	696
query77	399	374	315	315
query78	10937	11189	10434	10434
query79	1139	762	567	567
query80	868	635	570	570
query81	530	260	225	225
query82	1343	152	116	116
query83	341	260	245	245
query84	297	126	99	99
query85	912	498	455	455
query86	394	301	277	277
query87	3096	3123	3030	3030
query88	3515	2651	2640	2640
query89	428	367	346	346
query90	1881	181	178	178
query91	172	167	141	141
query92	78	76	68	68
query93	911	827	498	498
query94	507	327	286	286
query95	596	345	318	318
query96	643	514	227	227
query97	2488	2465	2390	2390
query98	236	218	218	218
query99	978	968	923	923
Total cold run time: 248283 ms
Total hot run time: 168166 ms

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 23, 2026

Cloud UT Coverage Report

Increment line coverage 🎉

Increment coverage report
Complete coverage report

Category Coverage
Function Coverage 78.63% (1796/2284)
Line Coverage 64.39% (32280/50130)
Region Coverage 65.32% (16172/24760)
Branch Coverage 55.74% (8615/15456)

@hello-stephen
Copy link
Contributor

hello-stephen commented Mar 23, 2026

FE UT Coverage Report

Increment line coverage 38.69% (785/2029) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@zclllyybb zclllyybb Awaiting requested review from zclllyybb zclllyybb will be requested when the pull request is marked ready for review zclllyybb is a code owner

@BiteTheDDDDt BiteTheDDDDt Awaiting requested review from BiteTheDDDDt BiteTheDDDDt will be requested when the pull request is marked ready for review BiteTheDDDDt is a code owner

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@kaka11chen @hello-stephen @doris-robot