Skip to content

[fix](fe) Fix invalid semi-join transpose when the bottom join is a mark join - #66574

Draft
starocean999 wants to merge 1 commit into
apache:masterfrom
starocean999:master_semi_reorder
Draft

[fix](fe) Fix invalid semi-join transpose when the bottom join is a mark join#66574
starocean999 wants to merge 1 commit into
apache:masterfrom
starocean999:master_semi_reorder

Conversation

@starocean999

Copy link
Copy Markdown
Contributor

What problem does this PR solve?

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:
A query with a subquery inside the ON condition of an anti join failed at
physical planning with "A expression contains slot not from children", e.g.:

SELECT t1.* FROM t1 LEFT ANTI JOIN t2
    ON t1.k2 = t2.k3 AND t1.k1 NOT IN (SELECT t3.k1 FROM t3 WHERE t1.k2 = t3.k2);

The subquery is unnested into a mark join (t1 LEFT ANTI JOIN t3) that produces
a mark slot, and the outer anti join references the mark slot in its conjuncts.
Root cause: the exploration rule SemiJoinSemiJoinTransposeProject transposes
two nested left semi/anti joins (A ⟕̸ B) ⟕̸ C into (A ⟕̸ C) ⟕̸ B. When the
bottom join (A ⟕̸ B) is a mark join, the transposed plan builds
newBottomSemi = (A ⟕̸ C) from the top join, inheriting the top join's conjuncts
that reference the bottom mark slot, while the mark slot is now produced above by
the new top mark join. The mark slot is therefore referenced by a join whose
children don't output it, and physical planning fails with "slot not from
children".

The fix rejects the transpose when the bottom semi join is a mark join and the
top semi join references the bottom mark slot in its conjuncts, so the mark join
is always kept below the join that consumes the mark slot. After the fix the
query above executes correctly and returns the expected result.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@starocean999

Copy link
Copy Markdown
Contributor Author

run buildall

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 28806 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit be41d10980d9bb66eee9979ab8e1b695f1a659a0, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17684	4032	3963	3963
q2	2057	311	207	207
q3	10256	1332	810	810
q4	4682	471	339	339
q5	7483	833	543	543
q6	186	179	144	144
q7	730	794	609	609
q8	9332	1527	1576	1527
q9	5865	4051	3999	3999
q10	6749	1633	1363	1363
q11	497	344	320	320
q12	721	579	460	460
q13	18115	3197	2765	2765
q14	266	259	245	245
q15	q16	743	731	659	659
q17	1030	955	1021	955
q18	6571	5619	5583	5583
q19	1151	1235	1076	1076
q20	849	683	580	580
q21	5515	2626	2366	2366
q22	412	352	293	293
Total cold run time: 100894 ms
Total hot run time: 28806 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4369	4345	4177	4177
q2	278	313	203	203
q3	4509	4946	4346	4346
q4	2132	2232	1422	1422
q5	4219	4144	4070	4070
q6	232	185	133	133
q7	1692	1604	1416	1416
q8	2617	2153	2015	2015
q9	7350	7167	7258	7167
q10	4321	4302	3847	3847
q11	568	409	393	393
q12	728	762	512	512
q13	3240	3572	3204	3204
q14	308	309	279	279
q15	q16	700	723	644	644
q17	1318	1307	1321	1307
q18	12198	10999	11842	10999
q19	1170	1163	1169	1163
q20	2236	2257	1939	1939
q21	5647	4862	4733	4733
q22	521	443	409	409
Total cold run time: 60353 ms
Total hot run time: 54378 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 159090 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit be41d10980d9bb66eee9979ab8e1b695f1a659a0, data reload: false

query5	4322	603	438	438
query6	460	213	205	205
query7	4870	569	350	350
query8	348	172	146	146
query9	8780	4039	4000	4000
query10	470	359	305	305
query11	5880	2191	1983	1983
query12	155	98	93	93
query13	1248	591	423	423
query14	6103	4260	4000	4000
query14_1	3830	3801	3793	3793
query15	203	196	184	184
query16	985	495	446	446
query17	929	725	569	569
query18	2436	477	343	343
query19	205	189	148	148
query20	103	114	103	103
query21	237	158	143	143
query22	13070	13020	12854	12854
query23	15779	15082	14705	14705
query23_1	14819	14618	14735	14618
query24	7646	1695	1231	1231
query24_1	1253	1263	1245	1245
query25	586	467	405	405
query26	1329	354	213	213
query27	2595	591	389	389
query28	4572	2044	2068	2044
query29	1107	633	525	525
query30	345	264	218	218
query31	1167	1107	1050	1050
query32	107	60	61	60
query33	510	306	241	241
query34	1192	1212	628	628
query35	732	742	635	635
query36	766	763	696	696
query37	153	114	96	96
query38	1831	1767	1694	1694
query39	834	831	805	805
query39_1	799	780	790	780
query40	245	160	141	141
query41	65	65	62	62
query42	94	91	92	91
query43	326	320	280	280
query44	1453	769	779	769
query45	187	172	172	172
query46	1042	1152	736	736
query47	1517	1551	1524	1524
query48	387	462	290	290
query49	613	408	295	295
query50	1055	411	382	382
query51	11044	10623	10731	10623
query52	87	85	73	73
query53	272	271	201	201
query54	286	245	230	230
query55	79	71	68	68
query56	322	301	268	268
query57	1014	1026	914	914
query58	283	272	248	248
query59	1595	1634	1446	1446
query60	315	270	254	254
query61	151	146	149	146
query62	410	320	271	271
query63	245	200	195	195
query64	2860	1046	841	841
query65	3927	3855	3829	3829
query66	1833	482	358	358
query67	20064	19900	19962	19900
query68	3112	1519	1060	1060
query69	407	292	266	266
query70	891	792	810	792
query71	363	335	317	317
query72	3059	2680	2416	2416
query73	848	731	427	427
query74	4624	4532	4330	4330
query75	2390	2352	2015	2015
query76	2322	1150	792	792
query77	358	383	319	319
query78	11287	11097	10620	10620
query79	1396	1172	756	756
query80	1235	552	466	466
query81	523	327	292	292
query82	672	178	140	140
query83	389	329	301	301
query84	340	170	130	130
query85	947	604	499	499
query86	413	235	224	224
query87	2015	1986	1837	1837
query88	3713	2804	2773	2773
query89	394	320	298	298
query90	1951	203	202	202
query91	199	194	170	170
query92	64	62	55	55
query93	1616	1504	1073	1073
query94	704	348	281	281
query95	785	616	492	492
query96	1127	888	350	350
query97	2457	2472	2364	2364
query98	200	186	180	180
query99	739	728	614	614
Total cold run time: 246208 ms
Total hot run time: 159090 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 24.05 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit be41d10980d9bb66eee9979ab8e1b695f1a659a0, data reload: false

query1	0.01	0.00	0.01
query2	0.09	0.05	0.06
query3	0.25	0.14	0.13
query4	1.61	0.14	0.14
query5	0.24	0.23	0.22
query6	1.17	0.83	0.77
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.38	0.31	0.32
query10	0.54	0.56	0.55
query11	0.21	0.13	0.15
query12	0.18	0.15	0.14
query13	0.48	0.46	0.47
query14	1.01	1.00	0.99
query15	0.61	0.59	0.59
query16	0.34	0.33	0.33
query17	1.11	1.14	1.12
query18	0.23	0.20	0.20
query19	2.15	1.94	2.00
query20	0.02	0.01	0.01
query21	15.42	0.22	0.14
query22	4.92	0.05	0.05
query23	16.14	0.30	0.13
query24	2.89	0.41	0.36
query25	0.11	0.06	0.05
query26	0.73	0.20	0.19
query27	0.05	0.05	0.04
query28	3.50	0.80	0.36
query29	12.58	4.03	3.19
query30	0.27	0.15	0.16
query31	2.77	0.56	0.32
query32	3.22	0.59	0.49
query33	3.24	3.15	3.15
query34	15.66	3.96	3.29
query35	3.23	3.22	3.22
query36	0.56	0.43	0.42
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.08	0.03	0.03
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.53 s
Total hot run time: 24.05 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 12.00% (9/75) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants