Skip to content

branch-3.0: [fix])(catalog)add equals for external table #47956#48348

Merged
dataroaring merged 1 commit intobranch-3.0from
auto-pick-47956-branch-3.0
Mar 10, 2025
Merged

branch-3.0: [fix])(catalog)add equals for external table #47956#48348
dataroaring merged 1 commit intobranch-3.0from
auto-pick-47956-branch-3.0

Conversation

@github-actions
Copy link
Copy Markdown
Contributor

Cherry-picked from #47956

### What problem does this PR solve?

The materialized view is created based on external table, and sometimes
refreshing the materialized view may report that the corresponding
partition cannot be found.

The reason is that when the materialized view is refreshed, predicates
are generated based on the refreshed partition, and the problematic plan
does not include predicates.

The code for adding predicates is as follows:
```
if (predicates.getPredicates().containsKey(table)) {
return new LogicalFilter<>(ImmutableSet.of(ExpressionUtils.or(predicates.getPredicates().get(table))),
unboundRelation);
}
```
If the table and predicates. get() are not the same object when setting
predicates, it will not be obtained, resulting in the absence of
predicates

PS: The table cache will refresh within 24 hours by default, and the
objects will change
@hello-stephen
Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring reopened this Feb 26, 2025
@hello-stephen
Copy link
Copy Markdown
Contributor

run buildall

@doris-robot
Copy link
Copy Markdown

TPC-H: Total hot run time: 40330 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 208c2d613f21d9519bc116b4664af16a13fe7273, data reload: false

------ Round 1 ----------------------------------
q1	17726	6821	6732	6732
q2	3526	182	159	159
q3	11157	1087	1193	1087
q4	10580	736	742	736
q5	7851	2923	2826	2826
q6	229	139	132	132
q7	985	624	611	611
q8	10057	1995	2061	1995
q9	9641	6467	6469	6467
q10	7088	2285	2355	2285
q11	2586	258	247	247
q12	504	224	221	221
q13	19540	2998	3070	2998
q14	234	205	213	205
q15	499	471	466	466
q16	682	604	600	600
q17	985	623	519	519
q18	7160	6652	6659	6652
q19	1409	1097	1112	1097
q20	429	215	204	204
q21	4261	3410	3142	3142
q22	1091	949	956	949
Total cold run time: 118220 ms
Total hot run time: 40330 ms

----- Round 2, with runtime_filter_mode=off -----
q1	6690	6626	6676	6626
q2	357	239	254	239
q3	3042	2954	3000	2954
q4	2013	1733	1765	1733
q5	5625	5715	5835	5715
q6	209	131	128	128
q7	2209	1807	1832	1807
q8	3388	3399	3516	3399
q9	8888	8771	8763	8763
q10	3532	3534	3544	3534
q11	605	490	492	490
q12	871	601	592	592
q13	9498	3144	3182	3144
q14	297	278	271	271
q15	515	467	463	463
q16	677	651	658	651
q17	1837	1593	1581	1581
q18	8169	7705	7601	7601
q19	1669	1530	1552	1530
q20	2122	1892	1850	1850
q21	5375	5352	5304	5304
q22	1153	996	1045	996
Total cold run time: 68741 ms
Total hot run time: 59371 ms

@doris-robot
Copy link
Copy Markdown

TPC-DS: Total hot run time: 196866 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 208c2d613f21d9519bc116b4664af16a13fe7273, data reload: false

query1	1335	943	912	912
query2	6251	2083	2120	2083
query3	10819	4342	4217	4217
query4	65266	29053	23265	23265
query5	4948	462	458	458
query6	401	167	176	167
query7	5661	308	308	308
query8	313	223	230	223
query9	9194	2590	2600	2590
query10	490	275	267	267
query11	17477	15188	15761	15188
query12	160	111	103	103
query13	1556	431	441	431
query14	10991	7366	8040	7366
query15	210	188	181	181
query16	6714	475	495	475
query17	1086	603	587	587
query18	1726	347	334	334
query19	237	166	173	166
query20	117	117	123	117
query21	206	109	104	104
query22	4705	4520	4573	4520
query23	34552	33768	33855	33768
query24	6174	2912	2945	2912
query25	551	439	433	433
query26	654	177	183	177
query27	1832	357	353	353
query28	4144	2415	2411	2411
query29	737	465	484	465
query30	252	165	162	162
query31	1016	818	822	818
query32	72	59	54	54
query33	434	283	291	283
query34	940	512	525	512
query35	859	740	742	740
query36	1080	982	1022	982
query37	119	67	70	67
query38	4123	4152	4009	4009
query39	1691	1496	1491	1491
query40	209	110	97	97
query41	49	48	47	47
query42	121	100	101	100
query43	549	510	504	504
query44	1178	812	840	812
query45	182	172	168	168
query46	1142	730	725	725
query47	2013	1950	1949	1949
query48	488	384	387	384
query49	724	390	399	390
query50	875	429	429	429
query51	7295	7266	7025	7025
query52	97	86	86	86
query53	250	183	179	179
query54	555	458	447	447
query55	81	77	75	75
query56	245	244	252	244
query57	1243	1105	1104	1104
query58	209	200	214	200
query59	3259	3149	2901	2901
query60	277	250	252	250
query61	112	103	110	103
query62	764	667	668	667
query63	218	188	188	188
query64	1360	663	640	640
query65	3262	3179	3201	3179
query66	713	298	302	298
query67	15906	15531	15506	15506
query68	4107	573	567	567
query69	427	257	261	257
query70	1224	1100	1161	1100
query71	359	256	251	251
query72	6367	4187	4089	4089
query73	745	350	348	348
query74	10253	9124	8957	8957
query75	3380	2655	2661	2655
query76	1832	1175	1041	1041
query77	500	271	282	271
query78	10627	9590	9460	9460
query79	1242	595	600	595
query80	842	433	435	433
query81	510	242	242	242
query82	1282	93	92	92
query83	252	141	152	141
query84	284	79	83	79
query85	877	305	308	305
query86	336	298	306	298
query87	4421	4254	4410	4254
query88	3689	2390	2364	2364
query89	416	292	295	292
query90	2017	189	189	189
query91	183	149	150	149
query92	61	51	48	48
query93	1261	527	535	527
query94	806	305	305	305
query95	355	269	262	262
query96	605	285	283	283
query97	3300	3158	3121	3121
query98	210	212	203	203
query99	1589	1297	1292	1292
Total cold run time: 316828 ms
Total hot run time: 196866 ms

@doris-robot
Copy link
Copy Markdown

ClickBench: Total hot run time: 32.85 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 208c2d613f21d9519bc116b4664af16a13fe7273, data reload: false

query1	0.03	0.03	0.03
query2	0.07	0.03	0.03
query3	0.24	0.06	0.07
query4	1.63	0.10	0.11
query5	0.51	0.51	0.52
query6	1.13	0.74	0.73
query7	0.02	0.02	0.01
query8	0.04	0.03	0.04
query9	0.57	0.51	0.52
query10	0.57	0.56	0.56
query11	0.14	0.10	0.10
query12	0.14	0.12	0.11
query13	0.62	0.60	0.59
query14	2.84	2.86	2.72
query15	0.89	0.82	0.83
query16	0.38	0.39	0.38
query17	1.07	1.04	1.01
query18	0.23	0.22	0.22
query19	1.93	1.85	1.99
query20	0.02	0.01	0.01
query21	15.37	0.59	0.58
query22	2.61	2.15	2.27
query23	16.91	0.94	0.90
query24	3.21	1.23	2.27
query25	0.28	0.11	0.09
query26	0.57	0.15	0.14
query27	0.04	0.05	0.05
query28	9.18	0.52	0.48
query29	12.58	3.25	3.24
query30	0.25	0.06	0.05
query31	2.86	0.40	0.38
query32	3.22	0.47	0.46
query33	3.02	2.98	3.00
query34	16.99	4.46	4.50
query35	4.49	4.51	4.48
query36	0.67	0.49	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.04	0.03	0.02
query40	0.16	0.13	0.13
query41	0.08	0.03	0.02
query42	0.03	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 105.8 s
Total hot run time: 32.85 s

Copy link
Copy Markdown
Contributor

@dataroaring dataroaring left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dataroaring dataroaring merged commit 2edecb8 into branch-3.0 Mar 10, 2025
23 of 24 checks passed
@github-actions github-actions bot deleted the auto-pick-47956-branch-3.0 branch March 10, 2025 09:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants