Skip to content

[fix](nereids) Fix use aggregate mv wrongly when rewrite query which only contains join #30858

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
morrySnow merged 6 commits into from
Feb 6, 2024
Merged

[fix](nereids) Fix use aggregate mv wrongly when rewrite query which only contains join #30858

morrySnow merged 6 commits into from
Feb 6, 2024

Conversation

@seawinde
Copy link
Contributor

@seawinde seawinde commented Feb 5, 2024

Proposed changes

the materialized view def is as following:

       select 
          o_orderdate, 
          o_shippriority, 
          o_comment, 
          l_orderkey, 
          o_orderkey, 
          count(*) 
        from 
          orders 
          left join lineitem on l_orderkey = o_orderkey
          group by o_orderdate, 
          o_shippriority, 
          o_comment, 
          l_orderkey;

the query should rewrite success by using above materialized view

        select 
          o_orderdate, 
          o_shippriority, 
          o_comment, 
          l_orderkey, 
          ps_partkey, 
          count(*) 
        from 
          orders left 
          join lineitem on l_orderkey = o_orderkey
          left join partsupp on ps_partkey = l_orderkey
          group by
         o_orderdate, 
         o_shippriority, 
         o_comment, 
         l_orderkey, 
         ps_partkey;

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

@seawinde seawinde force-pushed the fix_join_agg_rewrite_fail branch from 79f4614 to 4d373e5 Compare February 5, 2024 09:39
@seawinde
Copy link
Contributor Author

seawinde commented Feb 5, 2024

run buildall

@seawinde
Copy link
Contributor Author

seawinde commented Feb 5, 2024

run buildall

morrySnow
morrySnow previously approved these changes Feb 5, 2024
View reviewed changes
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added approved Indicates a PR has been approved by one committer. reviewed labels Feb 5, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 5, 2024

PR approved by anyone and no changes requested.

@seawinde
Copy link
Contributor Author

seawinde commented Feb 5, 2024

run buildall

@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Feb 5, 2024
@seawinde
Copy link
Contributor Author

seawinde commented Feb 5, 2024

run buildall

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

TPC-H: Total hot run time: 37503 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 7d0855bc51c7c8caa5cbcbb97ff28cf90edd876f, data reload: false

------ Round 1 ----------------------------------
q1	17681	4944	4325	4325
q2	2055	140	130	130
q3	10687	937	906	906
q4	4642	744	776	744
q5	7981	2817	2810	2810
q6	183	119	120	119
q7	1116	726	722	722
q8	9281	2008	2028	2008
q9	7330	6385	6388	6385
q10	8132	2470	2454	2454
q11	405	196	201	196
q12	712	267	287	267
q13	18264	3339	3348	3339
q14	272	246	253	246
q15	536	497	491	491
q16	471	410	402	402
q17	934	527	471	471
q18	6789	6320	6318	6318
q19	2608	1462	1428	1428
q20	1733	355	376	355
q21	7643	3079	3196	3079
q22	824	319	308	308
Total cold run time: 110279 ms
Total hot run time: 37503 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4470	4397	4383	4383
q2	322	240	236	236
q3	3094	2871	2815	2815
q4	1867	1699	1712	1699
q5	5164	5246	5179	5179
q6	195	113	113	113
q7	2137	1753	1751	1751
q8	3076	3211	3248	3211
q9	8362	8279	8275	8275
q10	5765	3570	3548	3548
q11	538	475	473	473
q12	747	577	566	566
q13	16919	3058	3083	3058
q14	282	250	255	250
q15	527	491	490	490
q16	523	469	470	469
q17	1801	1696	1698	1696
q18	7971	7591	7582	7582
q19	9373	1513	1525	1513
q20	2167	1920	1901	1901
q21	6990	4590	4539	4539
q22	556	469	479	469
Total cold run time: 82846 ms
Total hot run time: 54216 ms

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

TPC-DS: Total hot run time: 181604 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 7d0855bc51c7c8caa5cbcbb97ff28cf90edd876f, data reload: false

query1	915	342	328	328
query2	6558	1934	2003	1934
query3	6698	206	201	201
query4	31587	21999	21943	21943
query5	4288	354	438	354
query6	262	160	177	160
query7	4601	285	272	272
query8	254	173	187	173
query9	9238	2254	2239	2239
query10	405	226	204	204
query11	18206	15183	15239	15183
query12	123	77	75	75
query13	1617	402	415	402
query14	9481	7170	6744	6744
query15	245	194	187	187
query16	8155	259	245	245
query17	2030	543	506	506
query18	2103	269	258	258
query19	361	143	145	143
query20	78	75	77	75
query21	197	131	121	121
query22	5117	4776	4756	4756
query23	30344	30132	30034	30034
query24	10498	2711	2736	2711
query25	578	338	340	338
query26	1190	140	155	140
query27	2424	298	293	293
query28	7219	1838	1844	1838
query29	867	605	595	595
query30	287	133	138	133
query31	909	682	699	682
query32	88	52	50	50
query33	731	216	219	216
query34	1105	458	450	450
query35	863	763	772	763
query36	1020	896	914	896
query37	114	64	59	59
query38	3261	3155	3133	3133
query39	1312	1271	1239	1239
query40	189	96	91	91
query41	37	35	33	33
query42	103	86	86	86
query43	489	469	504	469
query44	1061	675	689	675
query45	190	179	177	177
query46	1054	648	649	648
query47	1571	1487	1513	1487
query48	433	340	347	340
query49	1075	293	291	291
query50	777	384	366	366
query51	5239	5185	5110	5110
query52	99	88	92	88
query53	342	266	260	260
query54	283	214	225	214
query55	83	76	76	76
query56	221	201	209	201
query57	955	929	874	874
query58	211	172	180	172
query59	2323	2294	2340	2294
query60	238	203	217	203
query61	83	83	83	83
query62	674	341	350	341
query63	297	267	268	267
query64	5194	3761	3492	3492
query65	3259	3250	3218	3218
query66	818	319	301	301
query67	14419	14114	14195	14114
query68	4691	518	555	518
query69	461	331	312	312
query70	1259	1212	1234	1212
query71	324	239	241	239
query72	6064	2835	2691	2691
query73	690	317	323	317
query74	6635	6259	6224	6224
query75	3101	2332	2291	2291
query76	3252	941	955	941
query77	367	226	223	223
query78	9192	8734	8698	8698
query79	2515	493	483	483
query80	1370	359	335	335
query81	534	197	196	196
query82	824	77	75	75
query83	253	131	123	123
query84	236	83	85	83
query85	1565	334	323	323
query86	477	314	312	312
query87	3436	3226	3249	3226
query88	3732	2298	2308	2298
query89	446	360	348	348
query90	2011	162	163	162
query91	150	123	122	122
query92	56	41	42	41
query93	2113	495	481	481
query94	1293	178	178	178
query95	467	7914	7860	7860
query96	590	279	272	272
query97	4210	4123	4145	4123
query98	203	189	198	189
query99	1003	664	698	664
Total cold run time: 280566 ms
Total hot run time: 181604 ms

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

ClickBench: Total hot run time: 30.88 s 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 7d0855bc51c7c8caa5cbcbb97ff28cf90edd876f, data reload: false

query1	0.04	0.03	0.04
query2	0.06	0.02	0.03
query3	0.22	0.06	0.06
query4	1.67	0.10	0.10
query5	0.53	0.52	0.51
query6	1.22	0.64	0.65
query7	0.01	0.01	0.01
query8	0.04	0.03	0.02
query9	0.55	0.50	0.49
query10	0.56	0.55	0.55
query11	0.12	0.09	0.09
query12	0.10	0.08	0.10
query13	0.61	0.61	0.60
query14	0.78	0.79	0.82
query15	0.78	0.77	0.78
query16	0.37	0.37	0.37
query17	0.95	0.95	1.02
query18	0.21	0.27	0.25
query19	1.82	1.82	1.76
query20	0.01	0.01	0.01
query21	15.42	0.58	0.54
query22	3.08	2.00	1.97
query23	17.37	0.82	0.82
query24	2.48	0.71	1.43
query25	0.37	0.28	0.06
query26	0.58	0.14	0.13
query27	0.06	0.04	0.06
query28	12.11	0.85	0.84
query29	12.53	3.24	3.32
query30	0.62	0.55	0.54
query31	2.78	0.34	0.35
query32	3.37	0.48	0.48
query33	3.19	3.25	3.23
query34	16.05	4.33	4.25
query35	4.34	4.30	4.40
query36	1.10	1.06	1.05
query37	0.06	0.05	0.05
query38	0.04	0.02	0.02
query39	0.02	0.01	0.02
query40	0.15	0.12	0.12
query41	0.07	0.02	0.01
query42	0.02	0.01	0.01
query43	0.02	0.02	0.02
Total cold run time: 106.48 s
Total hot run time: 30.88 s

@doris-robot
Copy link

doris-robot commented Feb 5, 2024

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 7d0855bc51c7c8caa5cbcbb97ff28cf90edd876f with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       13.7 seconds inserted 10000000 Rows, about 729K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Feb 6, 2024
@github-actions
Copy link
Contributor

github-actions bot commented Feb 6, 2024

PR approved by at least one committer and no changes requested.

@morrySnow morrySnow merged commit a270a18 into apache:master Feb 6, 2024
yiguolei pushed a commit that referenced this pull request Feb 6, 2024
@seawinde @Doris-Extras
[fix](nereids) Fix use aggregate mv wrongly when rewrite query which …
3ebffe4 
…only contains join (#30858)

the materialized view def is as following:
>            select 
>               o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey, 
>               o_orderkey, 
>               count(*) 
>             from 
>               orders 
>               left join lineitem on l_orderkey = o_orderkey
>               group by o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey;

the query should rewrite success by using above materialized view
>             select 
>               o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey, 
>               ps_partkey, 
>               count(*) 
>             from 
>               orders left 
>               join lineitem on l_orderkey = o_orderkey
>               left join partsupp on ps_partkey = l_orderkey
>               group by
>              o_orderdate, 
>              o_shippriority, 
>              o_comment, 
>              l_orderkey, 
>              ps_partkey;
yiguolei pushed a commit that referenced this pull request Feb 16, 2024
@seawinde @Doris-Extras
[fix](nereids) Fix use aggregate mv wrongly when rewrite query which …
5c2a4a8 
…only contains join (#30858)

the materialized view def is as following:
>            select 
>               o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey, 
>               o_orderkey, 
>               count(*) 
>             from 
>               orders 
>               left join lineitem on l_orderkey = o_orderkey
>               group by o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey;

the query should rewrite success by using above materialized view
>             select 
>               o_orderdate, 
>               o_shippriority, 
>               o_comment, 
>               l_orderkey, 
>               ps_partkey, 
>               count(*) 
>             from 
>               orders left 
>               join lineitem on l_orderkey = o_orderkey
>               left join partsupp on ps_partkey = l_orderkey
>               group by
>              o_orderdate, 
>              o_shippriority, 
>              o_comment, 
>              l_orderkey, 
>              ps_partkey;
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@morrySnow morrySnow morrySnow approved these changes

+1 more reviewer

@keanji-x keanji-x keanji-x approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

approved Indicates a PR has been approved by one committer. reviewed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@seawinde @doris-robot @keanji-x @morrySnow