Skip to content

branch-3.0: [opt](mtmv) support rewrite when mv has date_trunc but query doesn't have #44948#46295

Closed
github-actions[bot] wants to merge 1 commit intobranch-3.0from
auto-pick-44948-branch-3.0
Closed

branch-3.0: [opt](mtmv) support rewrite when mv has date_trunc but query doesn't have #44948#46295
github-actions[bot] wants to merge 1 commit intobranch-3.0from
auto-pick-44948-branch-3.0

Conversation

@github-actions
Copy link
Contributor

@github-actions github-actions bot commented Jan 2, 2025

Cherry-picked from #44948

@seawinde
[opt](mtmv) support rewrite when mv has date_trunc but query doesn't …
fa235d0 
…have (#44948)

### What problem does this PR solve?

Such as mv def is as following:

            select
              l_shipmode,
              date_trunc(l_commitdate, 'day') as day_trunc,
              count(*)
            from
              lineitem
            group by
              l_shipmode,
              date_trunc(l_commitdate, 'day');

query is as following, if query filter is like `l_commitdate >=
'2023-10-17' and l_commitdate < '2023-10-18'`
can be rewritten successfully by mv

            select
              l_shipmode,
              count(*)
            from
              lineitem
            where
              l_commitdate >= '2023-10-17' and  l_commitdate < '2023-10-18'
            group by
              l_shipmode;


### Release note

support rewrite when mv has date_trunc but query doesn't have
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 2, 2025

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@dataroaring dataroaring closed this Jan 2, 2025
@dataroaring dataroaring reopened this Jan 2, 2025
@hello-stephen
Copy link
Contributor

hello-stephen commented Jan 2, 2025

run buildall

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-H: Total hot run time: 40866 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fa235d0433249de27a3236f85bec79265db0d181, data reload: false

------ Round 1 ----------------------------------
q1	17561	7377	7253	7253
q2	2055	173	170	170
q3	10844	1058	1141	1058
q4	10516	711	754	711
q5	7775	2852	2830	2830
q6	234	144	142	142
q7	975	601	595	595
q8	9362	1935	1986	1935
q9	6511	6424	6362	6362
q10	7032	2322	2297	2297
q11	468	264	267	264
q12	400	211	205	205
q13	18014	2985	3057	2985
q14	248	221	217	217
q15	558	525	506	506
q16	675	609	616	609
q17	974	597	573	573
q18	7393	6863	6783	6783
q19	1437	1131	1038	1038
q20	464	211	209	209
q21	4067	3325	3093	3093
q22	1140	1031	1060	1031
Total cold run time: 108703 ms
Total hot run time: 40866 ms

----- Round 2, with runtime_filter_mode=off -----
q1	7227	7207	7235	7207
q2	325	232	227	227
q3	2914	2962	2962	2962
q4	2063	1843	1750	1750
q5	5696	5712	5727	5712
q6	224	138	136	136
q7	2250	1821	1840	1821
q8	3317	3572	3443	3443
q9	8871	8835	8807	8807
q10	3593	3545	3537	3537
q11	594	504	509	504
q12	833	651	596	596
q13	10451	3120	3120	3120
q14	300	270	274	270
q15	571	528	521	521
q16	709	676	653	653
q17	1843	1633	1593	1593
q18	8240	7868	7542	7542
q19	1676	1613	1597	1597
q20	2116	1925	1850	1850
q21	5585	5399	5315	5315
q22	1124	1068	1033	1033
Total cold run time: 70522 ms
Total hot run time: 60196 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

TPC-DS: Total hot run time: 197080 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fa235d0433249de27a3236f85bec79265db0d181, data reload: false

query1	1252	937	912	912
query2	6218	2022	2057	2022
query3	10937	4401	4352	4352
query4	66487	29292	23517	23517
query5	4922	451	446	446
query6	400	173	177	173
query7	5624	318	299	299
query8	311	234	229	229
query9	8996	2682	2672	2672
query10	463	271	262	262
query11	17397	15158	15738	15158
query12	148	99	100	99
query13	1511	445	445	445
query14	10329	7218	6872	6872
query15	214	190	181	181
query16	7109	507	504	504
query17	1222	580	582	580
query18	1816	326	318	318
query19	220	156	159	156
query20	119	114	110	110
query21	63	50	46	46
query22	4810	4560	4582	4560
query23	35012	34252	34171	34171
query24	6056	2861	2857	2857
query25	536	418	421	418
query26	666	171	167	167
query27	1720	311	312	311
query28	4375	2491	2476	2476
query29	731	462	456	456
query30	248	168	161	161
query31	1029	828	843	828
query32	63	55	53	53
query33	463	283	278	278
query34	910	515	509	509
query35	835	707	766	707
query36	1067	964	937	937
query37	121	73	73	73
query38	4053	3942	4053	3942
query39	1507	1475	1463	1463
query40	143	84	81	81
query41	50	48	47	47
query42	111	102	99	99
query43	540	490	504	490
query44	1197	824	810	810
query45	187	175	168	168
query46	1132	737	736	736
query47	2068	1896	1935	1896
query48	469	379	380	379
query49	726	375	393	375
query50	843	425	415	415
query51	7279	7260	7297	7260
query52	96	83	85	83
query53	251	175	177	175
query54	561	436	447	436
query55	75	73	75	73
query56	262	236	243	236
query57	1271	1148	1134	1134
query58	212	198	206	198
query59	3218	3037	2896	2896
query60	285	252	267	252
query61	110	103	103	103
query62	773	663	653	653
query63	207	183	191	183
query64	1426	653	623	623
query65	3250	3191	3217	3191
query66	700	300	328	300
query67	15943	15775	15650	15650
query68	4095	577	577	577
query69	433	268	273	268
query70	1202	1133	1045	1045
query71	361	263	258	258
query72	6454	4030	4064	4030
query73	745	345	347	345
query74	10266	8940	9069	8940
query75	3339	2619	2642	2619
query76	1867	1038	1001	1001
query77	505	269	277	269
query78	10495	9715	9603	9603
query79	1557	603	587	587
query80	893	425	425	425
query81	516	242	235	235
query82	1272	119	113	113
query83	229	139	143	139
query84	278	85	81	81
query85	889	290	285	285
query86	345	302	298	298
query87	4477	4242	4232	4232
query88	3814	2386	2337	2337
query89	404	288	282	282
query90	2025	182	187	182
query91	184	151	150	150
query92	65	50	54	50
query93	2033	545	550	545
query94	806	292	303	292
query95	356	259	257	257
query96	606	282	280	280
query97	3468	3239	3205	3205
query98	215	196	201	196
query99	1618	1304	1292	1292
Total cold run time: 319157 ms
Total hot run time: 197080 ms

@doris-robot
Copy link

doris-robot commented Jan 2, 2025

ClickBench: Total hot run time: 32.65 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fa235d0433249de27a3236f85bec79265db0d181, data reload: false

query1	0.04	0.03	0.02
query2	0.07	0.03	0.03
query3	0.23	0.07	0.06
query4	1.63	0.10	0.11
query5	0.53	0.53	0.52
query6	1.13	0.74	0.73
query7	0.02	0.02	0.02
query8	0.04	0.03	0.04
query9	0.60	0.50	0.52
query10	0.54	0.55	0.55
query11	0.13	0.10	0.11
query12	0.14	0.11	0.11
query13	0.60	0.59	0.59
query14	3.00	3.02	2.94
query15	0.91	0.84	0.82
query16	0.39	0.38	0.38
query17	1.05	1.06	1.04
query18	0.24	0.22	0.21
query19	1.89	1.93	1.99
query20	0.01	0.01	0.01
query21	15.35	0.62	0.57
query22	2.62	1.71	1.63
query23	16.95	1.05	0.86
query24	3.47	0.69	0.92
query25	0.27	0.10	0.05
query26	0.40	0.14	0.14
query27	0.04	0.05	0.05
query28	10.67	1.10	1.08
query29	12.58	3.23	3.19
query30	0.23	0.05	0.06
query31	2.88	0.39	0.37
query32	3.26	0.45	0.47
query33	2.94	3.03	3.00
query34	17.08	4.53	4.53
query35	4.57	4.55	4.51
query36	0.65	0.51	0.49
query37	0.10	0.06	0.06
query38	0.04	0.03	0.04
query39	0.03	0.02	0.02
query40	0.16	0.12	0.13
query41	0.07	0.02	0.02
query42	0.04	0.02	0.02
query43	0.03	0.03	0.03
Total cold run time: 107.62 s
Total hot run time: 32.65 s

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@hello-stephen @doris-robot @dataroaring @seawinde