[fix](iceberg) Invalidate related-table cache on external table refresh by suxiaogang223 · Pull Request #60867 · apache/doris

suxiaogang223 · 2026-02-27T03:25:45Z

What problem does this PR solve?

Issue Number: N/A

Related PR: #57972

Problem Summary:

In multi-FE deployments, Iceberg partition evolution (ADD/REPLACE/DROP PARTITION KEY) can leave some FEs with a stale
IcebergExternalTable.isValidRelatedTableCached value. This stale cache may cause incorrect partitioned/unpartitioned
judgment and query failures until the table is refreshed.

This PR:

Clears the related-table validation cache during OP_REFRESH_EXTERNAL_TABLE replay
(RefreshManager.refreshTableInternal()).
Removes the redundant per-operation manual cache clearing in IcebergMetadataOps.

Release note

None

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
- No need to test or manual test. Explain why:
  - Previous test can cover this change.
  - This is a refactor/code format and no logic has been changed.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

suxiaogang223 · 2026-02-27T03:25:56Z

run buildall

doris-robot · 2026-02-27T04:16:11Z

TPC-H: Total hot run time: 28934 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5cb860a68ef5e2e204e1412187e17f3b90186569, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17688	4488	4267	4267
q2	q3	10657	769	518	518
q4	4674	364	251	251
q5	7549	1180	1030	1030
q6	176	177	148	148
q7	779	856	669	669
q8	9303	1494	1308	1308
q9	4925	4770	4757	4757
q10	6806	1858	1657	1657
q11	480	264	239	239
q12	749	566	469	469
q13	17776	4234	3402	3402
q14	233	231	210	210
q15	923	816	795	795
q16	773	731	688	688
q17	697	859	435	435
q18	6044	5349	5364	5349
q19	1259	967	627	627
q20	511	505	401	401
q21	4966	2017	1453	1453
q22	385	311	261	261
Total cold run time: 97353 ms
Total hot run time: 28934 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4706	4541	4549	4541
q2	q3	1791	2240	1786	1786
q4	861	1181	807	807
q5	4055	4472	4345	4345
q6	185	204	171	171
q7	1778	1631	1521	1521
q8	2492	2658	2546	2546
q9	7483	7200	7379	7200
q10	2717	2901	2385	2385
q11	500	435	416	416
q12	490	572	444	444
q13	4069	4447	3682	3682
q14	282	298	267	267
q15	876	857	801	801
q16	718	766	713	713
q17	1200	1543	1326	1326
q18	6885	6825	6897	6825
q19	845	847	860	847
q20	2045	2184	2060	2060
q21	3972	3456	3341	3341
q22	469	460	381	381
Total cold run time: 48419 ms
Total hot run time: 46405 ms

doris-robot · 2026-02-27T04:27:36Z

TPC-DS: Total hot run time: 184203 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5cb860a68ef5e2e204e1412187e17f3b90186569, data reload: false

query5	4792	637	525	525
query6	329	228	216	216
query7	4227	468	274	274
query8	353	252	236	236
query9	8735	2737	2748	2737
query10	542	395	355	355
query11	17070	17420	17377	17377
query12	212	132	123	123
query13	1264	475	356	356
query14	6889	3375	3124	3124
query14_1	3087	2893	2958	2893
query15	216	198	197	197
query16	1008	503	562	503
query17	2076	742	647	647
query18	2871	487	376	376
query19	221	237	188	188
query20	145	139	132	132
query21	231	153	129	129
query22	5720	5665	4675	4675
query23	17359	16761	16610	16610
query23_1	16669	16634	16731	16634
query24	7118	1626	1260	1260
query24_1	1230	1238	1239	1238
query25	557	480	440	440
query26	1229	266	154	154
query27	2736	509	281	281
query28	4406	1860	1887	1860
query29	812	547	464	464
query30	311	246	207	207
query31	875	713	683	683
query32	80	75	69	69
query33	516	353	284	284
query34	909	915	567	567
query35	627	660	586	586
query36	1085	1126	944	944
query37	149	94	83	83
query38	2996	2896	2933	2896
query39	891	915	842	842
query39_1	849	822	829	822
query40	226	150	137	137
query41	62	65	58	58
query42	106	104	104	104
query43	369	381	346	346
query44	
query45	201	185	180	180
query46	866	993	599	599
query47	2147	2149	2049	2049
query48	310	310	226	226
query49	636	472	389	389
query50	690	278	213	213
query51	4120	4054	4123	4054
query52	103	108	98	98
query53	287	332	284	284
query54	295	276	265	265
query55	93	87	80	80
query56	324	346	322	322
query57	1345	1324	1296	1296
query58	294	323	276	276
query59	2540	2615	2592	2592
query60	333	328	318	318
query61	150	141	146	141
query62	639	596	532	532
query63	307	276	277	276
query64	4850	1274	978	978
query65	
query66	1389	454	354	354
query67	16331	16530	16737	16530
query68	
query69	401	339	313	313
query70	1034	1047	988	988
query71	341	307	291	291
query72	2816	2704	2406	2406
query73	538	539	334	334
query74	9995	9971	9771	9771
query75	2841	2738	2475	2475
query76	2294	1022	666	666
query77	365	380	304	304
query78	11146	11395	10700	10700
query79	3023	806	595	595
query80	1824	639	536	536
query81	589	292	254	254
query82	991	148	118	118
query83	334	254	247	247
query84	255	116	94	94
query85	896	482	418	418
query86	419	311	281	281
query87	3164	3111	3015	3015
query88	3502	2661	2665	2661
query89	423	368	352	352
query90	2046	175	172	172
query91	157	150	131	131
query92	77	77	68	68
query93	1353	834	497	497
query94	667	313	308	308
query95	599	398	321	321
query96	638	518	225	225
query97	2485	2490	2410	2410
query98	239	219	216	216
query99	1017	934	901	901
Total cold run time: 258495 ms
Total hot run time: 184203 ms

suxiaogang223 · 2026-02-27T06:20:41Z

run buildall

doris-robot · 2026-02-27T06:42:35Z

TPC-H: Total hot run time: 28695 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 5cb860a68ef5e2e204e1412187e17f3b90186569, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17658	4515	4318	4318
q2	q3	10647	772	520	520
q4	4680	351	253	253
q5	7552	1216	1008	1008
q6	182	179	153	153
q7	820	870	681	681
q8	9303	1456	1335	1335
q9	4790	4725	4686	4686
q10	6776	1865	1639	1639
q11	467	269	242	242
q12	693	568	466	466
q13	17749	4207	3420	3420
q14	221	230	215	215
q15	896	821	788	788
q16	724	727	676	676
q17	727	874	394	394
q18	6051	5382	5138	5138
q19	1235	998	635	635
q20	508	494	394	394
q21	4898	1989	1459	1459
q22	394	323	275	275
Total cold run time: 96971 ms
Total hot run time: 28695 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4666	4475	4592	4475
q2	q3	1773	2221	1780	1780
q4	863	1201	790	790
q5	4038	4383	4303	4303
q6	186	182	156	156
q7	1823	1655	1524	1524
q8	2617	2705	2521	2521
q9	7343	7378	7385	7378
q10	2617	2827	2426	2426
q11	504	426	414	414
q12	515	637	451	451
q13	3988	4410	3720	3720
q14	401	358	269	269
q15	846	807	804	804
q16	704	767	723	723
q17	1199	1589	1333	1333
q18	7059	6659	6502	6502
q19	935	865	917	865
q20	2128	2168	2078	2078
q21	4000	3608	3668	3608
q22	481	437	373	373
Total cold run time: 48686 ms
Total hot run time: 46493 ms

doris-robot · 2026-02-27T06:53:55Z

TPC-DS: Total hot run time: 183686 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 5cb860a68ef5e2e204e1412187e17f3b90186569, data reload: false

query5	5225	641	549	549
query6	344	245	221	221
query7	4217	478	277	277
query8	340	271	251	251
query9	8755	2746	2773	2746
query10	494	379	361	361
query11	17035	16690	16603	16603
query12	204	129	128	128
query13	1269	462	352	352
query14	6787	3196	2947	2947
query14_1	2785	2803	2794	2794
query15	206	196	183	183
query16	984	464	463	463
query17	1079	724	619	619
query18	2722	453	354	354
query19	211	209	184	184
query20	145	131	134	131
query21	223	142	121	121
query22	5374	5989	5568	5568
query23	17513	17115	16850	16850
query23_1	17075	16893	16670	16670
query24	7126	1670	1231	1231
query24_1	1242	1234	1238	1234
query25	575	506	471	471
query26	1219	258	145	145
query27	2788	487	282	282
query28	4458	1838	1857	1838
query29	800	566	464	464
query30	307	242	208	208
query31	865	724	620	620
query32	81	74	74	74
query33	509	331	286	286
query34	937	934	567	567
query35	660	669	642	642
query36	1108	1086	945	945
query37	139	92	82	82
query38	3005	2878	2850	2850
query39	893	856	848	848
query39_1	843	834	809	809
query40	233	150	138	138
query41	75	74	60	60
query42	104	101	104	101
query43	369	389	349	349
query44	
query45	200	190	223	190
query46	877	984	614	614
query47	2126	2175	2062	2062
query48	308	324	239	239
query49	633	459	377	377
query50	718	277	229	229
query51	4080	4093	4014	4014
query52	104	108	97	97
query53	287	333	286	286
query54	292	279	264	264
query55	91	87	82	82
query56	316	304	310	304
query57	1357	1314	1270	1270
query58	287	282	275	275
query59	2510	2641	2485	2485
query60	336	334	317	317
query61	150	149	147	147
query62	629	577	538	538
query63	311	280	286	280
query64	4864	1256	996	996
query65	
query66	1380	456	349	349
query67	16457	16412	16325	16325
query68	
query69	407	306	291	291
query70	942	901	969	901
query71	350	306	299	299
query72	2739	2651	2452	2452
query73	545	546	328	328
query74	10011	9893	9713	9713
query75	2835	2732	2445	2445
query76	2289	1028	678	678
query77	362	379	306	306
query78	11173	11456	10716	10716
query79	1184	802	593	593
query80	1380	610	532	532
query81	567	276	256	256
query82	983	148	114	114
query83	348	268	241	241
query84	253	114	97	97
query85	901	491	431	431
query86	439	342	310	310
query87	3136	3090	2983	2983
query88	3588	2691	2666	2666
query89	430	368	349	349
query90	1966	184	181	181
query91	168	154	133	133
query92	78	74	70	70
query93	956	815	496	496
query94	642	325	298	298
query95	576	334	399	334
query96	651	534	233	233
query97	2495	2541	2370	2370
query98	230	219	216	216
query99	999	981	934	934
Total cold run time: 254919 ms
Total hot run time: 183686 ms

github-actions · 2026-02-27T07:23:01Z

PR approved by at least one committer and no changes requested.

github-actions · 2026-02-27T07:23:04Z

PR approved by anyone and no changes requested.

…sh (#60867) Related PR: #57972 Problem Summary: In multi-FE deployments, Iceberg partition evolution (ADD/REPLACE/DROP PARTITION KEY) can leave some FEs with a stale `IcebergExternalTable.isValidRelatedTableCached` value. This stale cache may cause incorrect partitioned/unpartitioned judgment and query failures until the table is refreshed. This PR: - Clears the related-table validation cache during `OP_REFRESH_EXTERNAL_TABLE` replay (`RefreshManager.refreshTableInternal()`). - Removes the redundant per-operation manual cache clearing in `IcebergMetadataOps`.

… table refresh #60867 (#60885) Cherry-picked from #60867 Co-authored-by: Socrates <suyiteng@selectdb.com>

Invalidate Iceberg related-table cache on refresh

5cb860a

suxiaogang223 changed the title ~~Invalidate Iceberg related-table cache on refresh~~ [fix](iceberg) Invalidate related-table cache on external table refresh Feb 27, 2026

morningman added the dev/4.0.x label Feb 27, 2026

morningman approved these changes Feb 27, 2026

View reviewed changes

github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 27, 2026

github-actions bot added the reviewed label Feb 27, 2026

morningman approved these changes Feb 27, 2026

View reviewed changes

CalvinKirs approved these changes Feb 27, 2026

View reviewed changes

zhangstar333 approved these changes Feb 27, 2026

View reviewed changes

morningman merged commit e0578fe into apache:master Feb 27, 2026
35 of 38 checks passed

github-actions bot mentioned this pull request Feb 27, 2026

branch-4.0: [fix](iceberg) Invalidate related-table cache on external table refresh #60867 #60885

Merged

yiguolei added dev/4.0.4-merged and removed dev/4.0.x labels Feb 28, 2026

yiguolei pushed a commit that referenced this pull request Mar 1, 2026

branch-4.0: [fix](iceberg) Invalidate related-table cache on external…

3f86bec

… table refresh #60867 (#60885) Cherry-picked from #60867 Co-authored-by: Socrates <suyiteng@selectdb.com>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[fix](iceberg) Invalidate related-table cache on external table refresh#60867

[fix](iceberg) Invalidate related-table cache on external table refresh#60867
morningman merged 1 commit intoapache:masterfrom
suxiaogang223:iceberg-refresh-invalidate-related-cache

suxiaogang223 commented Feb 27, 2026 •

edited

Loading

Uh oh!

suxiaogang223 commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

suxiaogang223 commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

suxiaogang223 commented Feb 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

suxiaogang223 commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

suxiaogang223 commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

doris-robot commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

github-actions bot commented Feb 27, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

suxiaogang223 commented Feb 27, 2026 •

edited

Loading