Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[fix](nereids)semi join transpose rule produce wrong plan if there is mark join #39152

Merged
merged 2 commits into from
Aug 11, 2024

Conversation

starocean999
Copy link
Contributor

Proposed changes

Issue Number: close #xxx

@doris-robot
Copy link

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR

Since 2024-03-18, the Document has been moved to doris-website.
See Doris Document.

@starocean999
Copy link
Contributor Author

run buildall

@github-actions github-actions bot added the doing label Aug 9, 2024
@doris-robot
Copy link

TPC-H: Total hot run time: 38284 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 87a917fc76fbf02c7f125a7d48930c6b1936d251, data reload: false

------ Round 1 ----------------------------------
q1	18735	4434	4377	4377
q2	2731	175	178	175
q3	11168	1162	1090	1090
q4	10706	720	763	720
q5	8210	2560	2468	2468
q6	224	138	136	136
q7	956	592	588	588
q8	9309	2053	1894	1894
q9	8647	6540	6557	6540
q10	7031	2184	2268	2184
q11	472	245	244	244
q12	389	217	225	217
q13	18392	2983	2981	2981
q14	273	243	238	238
q15	533	475	493	475
q16	501	379	389	379
q17	976	693	667	667
q18	8055	7411	7487	7411
q19	5588	1059	1022	1022
q20	708	328	334	328
q21	4997	3119	3145	3119
q22	1077	1031	1040	1031
Total cold run time: 119678 ms
Total hot run time: 38284 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4476	4286	4276	4276
q2	375	278	271	271
q3	2868	2656	2654	2654
q4	1829	1646	1581	1581
q5	5230	5262	5216	5216
q6	218	132	128	128
q7	2047	1652	1701	1652
q8	3145	3311	3298	3298
q9	8359	8292	8281	8281
q10	3373	3193	3145	3145
q11	589	516	480	480
q12	765	590	596	590
q13	16650	3032	2985	2985
q14	295	277	270	270
q15	514	483	481	481
q16	457	419	419	419
q17	1782	1481	1484	1481
q18	7634	7474	7448	7448
q19	1652	1589	1611	1589
q20	1991	1759	1799	1759
q21	4880	4792	4747	4747
q22	1068	996	996	996
Total cold run time: 70197 ms
Total hot run time: 53747 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201913 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 87a917fc76fbf02c7f125a7d48930c6b1936d251, data reload: false

query1	930	369	352	352
query2	6456	1907	1756	1756
query3	6673	209	212	209
query4	34058	23331	23176	23176
query5	4196	484	476	476
query6	267	160	166	160
query7	4575	298	301	298
query8	245	205	199	199
query9	8515	2484	2447	2447
query10	533	491	443	443
query11	16970	15036	15142	15036
query12	141	103	98	98
query13	1645	392	362	362
query14	10216	7696	7569	7569
query15	268	218	216	216
query16	7750	535	442	442
query17	1741	614	556	556
query18	1969	293	292	292
query19	195	149	147	147
query20	115	107	112	107
query21	204	108	99	99
query22	4236	4049	4041	4041
query23	33936	33267	33250	33250
query24	11889	2613	2561	2561
query25	672	392	402	392
query26	1794	159	159	159
query27	2890	285	289	285
query28	7326	2022	2016	2016
query29	1153	431	422	422
query30	309	151	148	148
query31	965	748	776	748
query32	90	56	59	56
query33	758	291	280	280
query34	962	461	476	461
query35	961	815	807	807
query36	1080	941	951	941
query37	284	82	85	82
query38	4273	4095	4159	4095
query39	1451	1389	1367	1367
query40	279	122	117	117
query41	48	47	46	46
query42	121	102	103	102
query43	485	459	468	459
query44	1211	724	744	724
query45	239	210	217	210
query46	1105	714	739	714
query47	1868	1753	1755	1753
query48	374	304	312	304
query49	1210	461	428	428
query50	802	414	407	407
query51	6767	6886	6758	6758
query52	104	93	97	93
query53	256	201	181	181
query54	1025	463	459	459
query55	79	77	77	77
query56	290	259	256	256
query57	1153	1052	1072	1052
query58	248	231	236	231
query59	2923	2659	2662	2659
query60	287	261	272	261
query61	122	117	236	117
query62	852	630	643	630
query63	216	185	180	180
query64	10524	2270	1744	1744
query65	3211	3143	3129	3129
query66	1368	330	343	330
query67	15592	14714	14794	14714
query68	8894	550	565	550
query69	483	403	407	403
query70	1423	1132	1125	1125
query71	525	277	292	277
query72	20653	17232	17516	17232
query73	2215	329	328	328
query74	9179	8899	8792	8792
query75	5230	2642	2703	2642
query76	5355	1053	977	977
query77	761	346	298	298
query78	9729	8985	8921	8921
query79	7769	528	528	528
query80	2085	510	491	491
query81	613	225	218	218
query82	291	131	129	129
query83	298	146	142	142
query84	272	81	86	81
query85	1027	282	276	276
query86	352	280	300	280
query87	4739	4532	4574	4532
query88	4854	2479	2529	2479
query89	481	288	279	279
query90	2063	199	200	199
query91	127	99	95	95
query92	68	48	49	48
query93	5828	543	545	543
query94	1032	292	295	292
query95	354	257	261	257
query96	613	279	280	279
query97	3197	3060	3097	3060
query98	214	201	198	198
query99	1521	1278	1276	1276
Total cold run time: 337695 ms
Total hot run time: 201913 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.14 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 87a917fc76fbf02c7f125a7d48930c6b1936d251, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.04	0.04
query3	0.23	0.07	0.06
query4	1.66	0.08	0.08
query5	0.50	0.47	0.47
query6	1.13	0.73	0.73
query7	0.03	0.02	0.01
query8	0.05	0.05	0.05
query9	0.55	0.49	0.48
query10	0.54	0.55	0.54
query11	0.16	0.11	0.12
query12	0.15	0.13	0.12
query13	0.61	0.59	0.59
query14	0.76	0.78	0.81
query15	0.85	0.83	0.82
query16	0.37	0.36	0.37
query17	1.04	1.05	1.06
query18	0.24	0.22	0.23
query19	1.79	1.74	1.74
query20	0.02	0.03	0.01
query21	15.40	0.75	0.65
query22	3.99	8.21	1.40
query23	18.31	1.44	1.31
query24	2.21	0.25	0.22
query25	0.15	0.08	0.08
query26	0.29	0.22	0.21
query27	0.45	0.23	0.22
query28	13.22	1.03	1.00
query29	12.63	3.30	3.25
query30	0.24	0.05	0.05
query31	2.90	0.41	0.39
query32	3.25	0.47	0.49
query33	2.89	2.92	2.84
query34	16.90	4.33	4.35
query35	4.48	4.44	4.39
query36	0.65	0.48	0.45
query37	0.19	0.16	0.16
query38	0.15	0.15	0.15
query39	0.06	0.03	0.04
query40	0.15	0.12	0.12
query41	0.11	0.04	0.06
query42	0.05	0.04	0.05
query43	0.04	0.04	0.04
Total cold run time: 109.53 s
Total hot run time: 30.14 s

@starocean999
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 39536 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 0c89fbcaf06071d35d419692da7127714ebff4c8, data reload: false

------ Round 1 ----------------------------------
q1	17619	4336	4287	4287
q2	2021	176	173	173
q3	10477	1216	1057	1057
q4	10154	707	665	665
q5	7520	2520	2493	2493
q6	224	135	136	135
q7	966	593	594	593
q8	9226	1898	1930	1898
q9	8777	6597	6570	6570
q10	7058	2214	2239	2214
q11	454	251	247	247
q12	480	222	214	214
q13	17756	3005	2935	2935
q14	280	234	234	234
q15	535	484	479	479
q16	502	393	392	392
q17	964	674	721	674
q18	8002	7491	7373	7373
q19	4072	1013	1052	1013
q20	702	324	335	324
q21	5385	4580	4620	4580
q22	1112	986	1014	986
Total cold run time: 114286 ms
Total hot run time: 39536 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4411	4270	4247	4247
q2	388	274	266	266
q3	2854	2619	2613	2613
q4	1994	1798	1798	1798
q5	5507	5563	5438	5438
q6	226	136	130	130
q7	2135	1740	1788	1740
q8	3307	3463	3407	3407
q9	8765	8752	8944	8752
q10	3494	3301	3292	3292
q11	609	495	508	495
q12	804	610	653	610
q13	16066	3167	3203	3167
q14	340	281	300	281
q15	531	480	488	480
q16	496	429	475	429
q17	1848	1526	1523	1523
q18	8068	8132	7915	7915
q19	1768	1483	1714	1483
q20	2147	1931	1891	1891
q21	12271	5355	5412	5355
q22	1094	1012	1096	1012
Total cold run time: 79123 ms
Total hot run time: 56324 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 201766 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 0c89fbcaf06071d35d419692da7127714ebff4c8, data reload: false

query1	925	375	369	369
query2	6442	1906	1807	1807
query3	6646	207	217	207
query4	31215	23207	23050	23050
query5	3584	494	486	486
query6	293	174	171	171
query7	4569	311	293	293
query8	248	198	203	198
query9	8361	2499	2424	2424
query10	515	465	424	424
query11	17549	15010	14830	14830
query12	125	96	99	96
query13	1646	365	357	357
query14	9486	7688	7680	7680
query15	275	207	210	207
query16	7039	472	477	472
query17	1308	572	560	560
query18	1897	280	276	276
query19	200	162	136	136
query20	111	105	103	103
query21	204	104	102	102
query22	4634	4588	4470	4470
query23	34181	33594	33804	33594
query24	9836	2768	2649	2649
query25	595	402	380	380
query26	713	158	163	158
query27	2447	288	291	288
query28	6116	2041	2025	2025
query29	801	404	401	401
query30	251	150	148	148
query31	1016	789	756	756
query32	94	58	56	56
query33	619	277	271	271
query34	869	449	476	449
query35	932	828	819	819
query36	1080	892	929	892
query37	136	78	82	78
query38	4220	4201	4170	4170
query39	1427	1402	1408	1402
query40	204	117	114	114
query41	46	44	43	43
query42	112	96	96	96
query43	507	472	471	471
query44	1092	723	724	723
query45	227	204	211	204
query46	1085	744	703	703
query47	1858	1779	1797	1779
query48	369	292	290	290
query49	815	428	424	424
query50	795	407	405	405
query51	6708	6722	6694	6694
query52	96	95	84	84
query53	258	180	184	180
query54	847	461	452	452
query55	78	75	75	75
query56	274	252	251	251
query57	1131	1067	1043	1043
query58	235	244	241	241
query59	2830	2584	2579	2579
query60	286	253	255	253
query61	97	95	94	94
query62	793	656	633	633
query63	205	184	175	175
query64	9201	2292	1750	1750
query65	3212	3151	3138	3138
query66	819	329	403	329
query67	15238	14958	14864	14864
query68	4512	550	542	542
query69	407	359	382	359
query70	1198	1137	1141	1137
query71	388	280	271	271
query72	18006	16508	16539	16508
query73	759	323	325	323
query74	9040	8746	8738	8738
query75	3330	2625	2699	2625
query76	2122	1078	1002	1002
query77	433	313	312	312
query78	9423	9170	8954	8954
query79	2994	529	511	511
query80	2062	494	498	494
query81	604	219	220	219
query82	945	138	137	137
query83	321	142	150	142
query84	268	78	77	77
query85	1391	287	323	287
query86	466	316	358	316
query87	4739	4680	4522	4522
query88	4503	2482	2478	2478
query89	402	285	286	285
query90	1776	198	195	195
query91	119	96	93	93
query92	57	47	50	47
query93	4746	541	530	530
query94	847	301	296	296
query95	354	260	269	260
query96	630	279	277	277
query97	3311	3106	3052	3052
query98	232	200	190	190
query99	1618	1248	1246	1246
Total cold run time: 304026 ms
Total hot run time: 201766 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 30.69 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 0c89fbcaf06071d35d419692da7127714ebff4c8, data reload: false

query1	0.05	0.04	0.04
query2	0.09	0.05	0.05
query3	0.23	0.05	0.04
query4	1.68	0.08	0.07
query5	0.50	0.47	0.49
query6	1.13	0.73	0.73
query7	0.02	0.02	0.01
query8	0.05	0.05	0.04
query9	0.56	0.48	0.51
query10	0.53	0.55	0.55
query11	0.17	0.11	0.11
query12	0.15	0.13	0.12
query13	0.60	0.60	0.58
query14	0.76	0.78	0.79
query15	0.85	0.81	0.82
query16	0.38	0.37	0.36
query17	1.00	1.01	1.05
query18	0.23	0.21	0.22
query19	1.84	1.72	1.68
query20	0.01	0.00	0.01
query21	15.39	0.77	0.67
query22	4.12	7.63	1.89
query23	18.68	1.32	1.32
query24	2.06	0.21	0.22
query25	0.14	0.09	0.08
query26	0.29	0.21	0.21
query27	0.45	0.23	0.22
query28	13.41	1.02	0.99
query29	12.57	3.30	3.31
query30	0.23	0.05	0.05
query31	2.91	0.39	0.39
query32	3.24	0.48	0.48
query33	2.89	2.95	2.94
query34	16.96	4.34	4.39
query35	4.42	4.41	4.45
query36	0.64	0.48	0.48
query37	0.19	0.15	0.15
query38	0.15	0.15	0.15
query39	0.04	0.04	0.03
query40	0.15	0.13	0.12
query41	0.09	0.04	0.04
query42	0.06	0.05	0.04
query43	0.04	0.04	0.04
Total cold run time: 109.95 s
Total hot run time: 30.69 s

@starocean999 starocean999 marked this pull request as ready for review August 10, 2024 05:37
Copy link
Contributor

PR approved by anyone and no changes requested.

Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Aug 11, 2024
@starocean999 starocean999 merged commit 46fceed into apache:master Aug 11, 2024
32 checks passed
wyxxxcat pushed a commit to wyxxxcat/doris that referenced this pull request Aug 14, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by one committer. dev/3.0.1-merged not-merge/2.1 reviewed
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants