Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first #32633

Merged
merged 2 commits into from Mar 22, 2024
Merged

[fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first #32633

merged 2 commits into from Mar 22, 2024

Conversation

englefly
Copy link
Contributor

@englefly englefly commented Mar 21, 2024 •
edited

do not push down topn-filter through right/full outer join if the first orderkey is nulls first

Proposed changes

Issue Number: close #xxx

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@englefly
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 37873 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit b2c0dcaa573c25a273ef71371fea710f9be53fa2, data reload: false

------ Round 1 ----------------------------------
q1	17715	4668	4260	4260
q2	2529	160	153	153
q3	11154	1101	1180	1101
q4	10566	759	790	759
q5	7573	3086	3029	3029
q6	211	128	125	125
q7	1042	604	594	594
q8	9688	2033	2027	2027
q9	7183	6601	6555	6555
q10	9704	3345	3482	3345
q11	445	226	221	221
q12	384	204	199	199
q13	17784	2862	2836	2836
q14	243	211	213	211
q15	497	450	455	450
q16	460	379	358	358
q17	961	550	509	509
q18	7253	6564	6387	6387
q19	2225	1381	1439	1381
q20	545	254	263	254
q21	3528	2807	3022	2807
q22	354	312	312	312
Total cold run time: 112044 ms
Total hot run time: 37873 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4138	4080	4100	4080
q2	324	229	221	221
q3	3001	2879	2872	2872
q4	1876	1571	1544	1544
q5	5260	5266	5260	5260
q6	198	117	120	117
q7	2243	1887	1858	1858
q8	3155	3326	3308	3308
q9	8591	8611	8529	8529
q10	3695	3723	3720	3720
q11	541	455	437	437
q12	744	562	555	555
q13	16895	2850	2845	2845
q14	270	247	247	247
q15	481	453	454	453
q16	450	430	437	430
q17	1745	1505	1456	1456
q18	7492	7198	7143	7143
q19	1615	1513	1483	1483
q20	1922	1729	1735	1729
q21	4865	4594	4799	4594
q22	529	459	452	452
Total cold run time: 70030 ms
Total hot run time: 53333 ms

@englefly englefly changed the title [fix](Nereids) do not push down topn-filter through right outer join if the first or… [fix](Nereids) do not push down topn-filter through right outer join if the first orderkey is nulls first Mar 21, 2024
@doris-robot
Copy link

TPC-DS: Total hot run time: 181866 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit b2c0dcaa573c25a273ef71371fea710f9be53fa2, data reload: false

query1	934	357	337	337
query2	7305	2048	1844	1844
query3	6712	213	218	213
query4	31137	20639	20723	20639
query5	4259	410	394	394
query6	273	174	165	165
query7	4623	295	290	290
query8	222	170	169	169
query9	9849	2286	2283	2283
query10	480	240	238	238
query11	17572	14303	14213	14213
query12	136	98	84	84
query13	1629	419	426	419
query14	12204	11349	10710	10710
query15	240	206	190	190
query16	8025	261	265	261
query17	1921	563	546	546
query18	2044	305	297	297
query19	239	163	157	157
query20	91	87	91	87
query21	202	135	130	130
query22	4729	4492	4449	4449
query23	31909	31279	31305	31279
query24	10878	2824	2866	2824
query25	654	370	383	370
query26	1244	154	149	149
query27	3064	361	345	345
query28	8139	1873	1905	1873
query29	889	654	627	627
query30	304	144	145	144
query31	962	717	745	717
query32	101	56	56	56
query33	768	342	252	252
query34	995	492	484	484
query35	837	611	611	611
query36	992	857	861	857
query37	113	75	74	74
query38	3507	3450	3444	3444
query39	1452	1378	1376	1376
query40	206	117	112	112
query41	49	45	46	45
query42	108	96	92	92
query43	486	453	439	439
query44	1129	722	710	710
query45	271	262	251	251
query46	1064	673	678	673
query47	1669	1593	1613	1593
query48	429	346	354	346
query49	1126	335	334	334
query50	776	371	370	370
query51	6669	6635	6563	6563
query52	105	94	95	94
query53	337	275	275	275
query54	304	231	234	231
query55	87	76	81	76
query56	252	232	221	221
query57	1076	1009	1010	1009
query58	228	205	195	195
query59	2762	2546	2572	2546
query60	264	254	251	251
query61	92	92	91	91
query62	581	408	390	390
query63	298	277	278	277
query64	5546	3853	3803	3803
query65	3045	3006	3001	3001
query66	863	353	332	332
query67	14801	14560	14681	14560
query68	5422	512	514	512
query69	566	382	372	372
query70	1277	1153	1237	1153
query71	434	285	278	278
query72	6679	2815	2642	2642
query73	713	314	324	314
query74	7211	6579	6611	6579
query75	3486	2860	2906	2860
query76	3471	941	925	925
query77	423	273	271	271
query78	10361	9794	9767	9767
query79	8818	519	522	519
query80	2070	404	416	404
query81	561	209	213	209
query82	1109	205	203	203
query83	323	151	147	147
query84	295	77	79	77
query85	1693	326	314	314
query86	490	305	308	305
query87	3697	3486	3603	3486
query88	5065	2298	2286	2286
query89	552	357	371	357
query90	1996	178	174	174
query91	157	134	154	134
query92	64	45	48	45
query93	6944	494	485	485
query94	1219	174	173	173
query95	427	331	325	325
query96	602	265	265	265
query97	3065	2882	2865	2865
query98	227	211	210	210
query99	1060	733	759	733
Total cold run time: 306734 ms
Total hot run time: 181866 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit b2c0dcaa573c25a273ef71371fea710f9be53fa2 with default session variables
Stream load json:         18 seconds loaded 2358488459 Bytes, about 124 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       20.7 seconds inserted 10000000 Rows, about 483K ops/s

starocean999
starocean999 previously approved these changes Mar 22, 2024
View reviewed changes
@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions GitHub Actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@englefly
Copy link
Contributor Author

run buildall

@englefly englefly changed the title [fix](Nereids) do not push down topn-filter through right outer join if the first orderkey is nulls first [fix](Nereids) do not push down topn-filter through right/full outer join if the first orderkey is nulls first Mar 22, 2024
@github-actions github-actions bot removed the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 37345 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3043716893149a6c70563e573c123da7ef1f2583, data reload: false

------ Round 1 ----------------------------------
q1	17648	4168	4111	4111
q2	2029	152	146	146
q3	10588	1092	1232	1092
q4	10240	749	738	738
q5	7461	2986	2926	2926
q6	204	127	129	127
q7	1004	571	565	565
q8	9353	1993	1951	1951
q9	7110	6502	6395	6395
q10	8400	3282	3447	3282
q11	430	224	223	223
q12	446	198	195	195
q13	17787	2833	2862	2833
q14	229	197	208	197
q15	512	468	456	456
q16	475	361	360	360
q17	941	554	584	554
q18	7084	6481	6517	6481
q19	3888	1343	1400	1343
q20	537	261	256	256
q21	3536	2821	2953	2821
q22	354	293	300	293
Total cold run time: 110256 ms
Total hot run time: 37345 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4158	4102	4075	4075
q2	326	230	227	227
q3	2936	2771	2761	2761
q4	1831	1520	1538	1520
q5	5247	5440	5199	5199
q6	203	119	119	119
q7	2253	1855	1846	1846
q8	3165	3279	3283	3279
q9	8508	8496	8494	8494
q10	3693	3654	3636	3636
q11	551	459	452	452
q12	736	557	536	536
q13	16928	2849	2848	2848
q14	286	245	268	245
q15	495	457	456	456
q16	469	413	429	413
q17	1737	1504	1474	1474
q18	7535	7372	7156	7156
q19	1578	1470	1496	1470
q20	1918	1714	1703	1703
q21	4836	4630	4823	4630
q22	520	437	436	436
Total cold run time: 69909 ms
Total hot run time: 52975 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 180345 ms 
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3043716893149a6c70563e573c123da7ef1f2583, data reload: false

query1	943	358	347	347
query2	7502	2018	1952	1952
query3	6710	215	219	215
query4	30886	20498	20842	20498
query5	4316	407	402	402
query6	284	178	175	175
query7	4636	296	288	288
query8	233	164	174	164
query9	8923	2287	2299	2287
query10	458	248	248	248
query11	15338	14115	14080	14080
query12	137	90	86	86
query13	1632	428	411	411
query14	11345	11147	10052	10052
query15	267	206	195	195
query16	8185	260	258	258
query17	1940	578	546	546
query18	2124	282	279	279
query19	347	166	164	164
query20	89	91	86	86
query21	203	131	131	131
query22	4672	4485	4437	4437
query23	31719	30960	31017	30960
query24	10656	2830	2809	2809
query25	583	360	365	360
query26	1162	150	152	150
query27	2417	346	350	346
query28	7187	1867	1879	1867
query29	854	636	602	602
query30	298	144	153	144
query31	970	746	732	732
query32	92	55	53	53
query33	760	264	319	264
query34	1004	473	480	473
query35	810	601	605	601
query36	941	877	851	851
query37	122	75	75	75
query38	3578	3469	3403	3403
query39	1405	1378	1350	1350
query40	210	109	107	107
query41	49	45	45	45
query42	99	93	93	93
query43	486	455	449	449
query44	1166	725	718	718
query45	271	251	248	248
query46	1084	701	689	689
query47	1670	1598	1585	1585
query48	443	362	342	342
query49	1131	338	321	321
query50	760	365	367	365
query51	6697	6568	6591	6568
query52	109	88	90	88
query53	340	275	279	275
query54	299	254	254	254
query55	85	82	76	76
query56	249	233	241	233
query57	1094	997	1000	997
query58	225	205	209	205
query59	2796	2605	2593	2593
query60	284	260	263	260
query61	118	117	116	116
query62	608	405	407	405
query63	304	280	280	280
query64	5703	4016	4018	4016
query65	3055	2986	2998	2986
query66	905	358	349	349
query67	15556	14629	14309	14309
query68	6935	520	523	520
query69	635	391	374	374
query70	1184	1166	1098	1098
query71	501	288	278	278
query72	6468	2812	2592	2592
query73	729	314	317	314
query74	8029	6580	6607	6580
query75	4076	2785	2789	2785
query76	4440	868	883	868
query77	608	298	261	261
query78	10396	9745	9583	9583
query79	8574	521	512	512
query80	1702	389	402	389
query81	526	213	225	213
query82	1312	204	204	204
query83	217	147	150	147
query84	287	81	78	78
query85	1516	345	321	321
query86	464	299	304	299
query87	3720	3516	3554	3516
query88	4785	2323	2296	2296
query89	520	377	371	371
query90	1988	174	174	174
query91	170	148	137	137
query92	61	45	49	45
query93	6822	505	484	484
query94	1160	179	174	174
query95	432	332	344	332
query96	602	268	274	268
query97	3043	2878	2868	2868
query98	228	212	208	208
query99	1112	743	757	743
Total cold run time: 304766 ms
Total hot run time: 180345 ms

@doris-robot
Copy link

Load test result on machine: 'aliyun_ecs.c7a.8xlarge_32C64G'

Load test result on commit 3043716893149a6c70563e573c123da7ef1f2583 with default session variables
Stream load json:         19 seconds loaded 2358488459 Bytes, about 118 MB/s
Stream load orc:          59 seconds loaded 1101869774 Bytes, about 17 MB/s
Stream load parquet:      31 seconds loaded 861443392 Bytes, about 26 MB/s
Insert into select:       22.1 seconds inserted 10000000 Rows, about 452K ops/s

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Mar 22, 2024
@github-actions GitHub Actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@englefly englefly merged commit c8976d2 into apache:master Mar 22, 2024
29 of 33 checks passed
@englefly englefly deleted the topn-rightOuterJoin branch March 22, 2024 12:53
yiguolei pushed a commit that referenced this pull request Mar 24, 2024
@englefly @Doris-Extras
[fix](Nereids) do not push down topn-filter through right/full outer …
f62cdec 
…join if the first orderkey is nulls first (#32633)

* do not push down topn-filter through right/full outer join if the first order key is nulls first
@yiguolei yiguolei mentioned this pull request Mar 24, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@starocean999 starocean999 starocean999 approved these changes

@morrySnow morrySnow morrySnow approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by one committer. reviewed
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
4 participants
@englefly @doris-robot @starocean999 @morrySnow