Skip to content

[fix](nullable) Do not generate NULL-only access path for NOT NULL column made nullable by outer join - #66380

Closed
yujun777 wants to merge 1 commit into
apache:masterfrom
yujun777:fix-doris-27274-nested-nullable
Closed

[fix](nullable) Do not generate NULL-only access path for NOT NULL column made nullable by outer join#66380
yujun777 wants to merge 1 commit into
apache:masterfrom
yujun777:fix-doris-27274-nested-nullable

Conversation

@yujun777

@yujun777 yujun777 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor

Problem

After an outer join, IS NULL on a physical NOT NULL column (e.g. the right-side dimension column of a LEFT JOIN) crashes the BE:

[E-7412] Check failed: is_column_nullable(*dst)

Core stack:

doris::segment_v2::FileColumnIterator::read_by_rowids          column_reader.cpp
doris::segment_v2::SegmentIterator::_read_columns_by_rowids    segment_iterator.cpp
doris::segment_v2::SegmentIterator::_next_batch_internal       segment_iterator.cpp:2997

doris_main.cpp:518 logs Failed to initialize JNI / SIGABRT with the same Check failed when the query runs.

Root cause

Introduced by #62304 (commit 85ede0bafba0, "opt Optimize I/O operations for the IS NULL predicate"). That PR makes col IS NULL generate a col.NULL access path so BE reads only the null map (NULL_MAP_ONLY mode), saving I/O. The premise is that the column is physically nullable (has a null bitmap).

The bug: the rule only checks expression nullability (arg.nullable()). A LEFT JOIN makes the right-side NOT NULL column's output slot nullable, so arg.nullable() == true and a [col, NULL] access path is generated even though the physical column has no null bitmap. BE then reads the null map by rowids on a NOT NULL column and hits the is_column_nullable(*dst) check — the backend aborts, failing the user's base-table query.

This is unrelated to the IVM branch that first surfaced it: any ordinary base-table LEFT JOIN query with enable_prune_nested_column = true (default) reproduces the crash.

Fix

In AccessPathExpressionCollector.visitIsNull, skip the NULL-only access path generation when the SlotReference's physical column is NOT NULL (only made nullable by the outer join). The join output stays nullable and IS NULL is evaluated on the regular data path; no access info is sent to BE for the whole-column read (shouldSkipAccessInfo drops the single [col] path).

Also merged the existing sub-column-path early return into the same guard.

Verification

  • PruneNestedColumnTest.testIsNullOnNotNullColumnAfterLeftJoin: asserts no [segment, NULL] access path is generated after LEFT JOIN (50 tests, all pass)
  • Regression left_join_not_null_column.groovy: LEFT JOIN aggregate query returns correct result, BE stays alive; explain asserts no segment.NULL in the plan
  • Before the fix, the same regression crashes the BE with the exact stack above (reproduced locally with a 3-row fact table)

…umn made nullable by outer join

IS NULL predicate on a physical NOT NULL column that is only made nullable by an
outer join generates a [col, NULL] access path, because the rule only checks the
expression nullability. The column has no physical null bitmap, so BE crashes with
`Check failed: is_column_nullable(*dst)` in FileColumnIterator::read_by_rowids when
reading the null map by rowids.

Key changes:
- AccessPathExpressionCollector.visitIsNull: skip NULL-only access path when the
  SlotReference's physical column is NOT NULL (made nullable by outer join); the
  regular data path is sufficient to evaluate IS NULL after the join
- merge the sub-column-path early return into the same guard

Unit Test:
- PruneNestedColumnTest.testIsNullOnNotNullColumnAfterLeftJoin
- regression-test/suites/nereids_rules_p0/column_pruning/left_join_not_null_column.groovy
@hello-stephen

Copy link
Copy Markdown
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@yujun777

yujun777 commented Aug 3, 2026

Copy link
Copy Markdown
Contributor Author

run buildall

@morrySnow morrySnow changed the title [fix](nereids) Do not generate NULL-only access path for NOT NULL column made nullable by outer join [fix](nullable) Do not generate NULL-only access path for NOT NULL column made nullable by outer join Aug 3, 2026
@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-H: Total hot run time: 29086 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 1c7efc9ebb7ed4e2cc2c5c4c7ffc89d443be2e25, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17700	4133	4126	4126
q2	2053	330	209	209
q3	10317	1434	803	803
q4	4685	476	337	337
q5	7498	843	566	566
q6	190	182	142	142
q7	769	791	599	599
q8	9342	1598	1571	1571
q9	5308	4070	4090	4070
q10	6737	1624	1358	1358
q11	508	364	324	324
q12	716	596	443	443
q13	18108	3331	2716	2716
q14	268	260	241	241
q15	q16	738	728	667	667
q17	1034	998	935	935
q18	7264	5750	5615	5615
q19	1301	1365	1023	1023
q20	790	681	575	575
q21	6129	2694	2466	2466
q22	427	365	300	300
Total cold run time: 101882 ms
Total hot run time: 29086 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4378	4248	4329	4248
q2	289	346	219	219
q3	4559	4911	4426	4426
q4	2220	2260	1419	1419
q5	4327	4166	4206	4166
q6	229	174	134	134
q7	1717	1637	2008	1637
q8	2758	2318	2338	2318
q9	7615	7618	7348	7348
q10	4305	4341	3907	3907
q11	620	446	380	380
q12	713	736	529	529
q13	3199	3454	2920	2920
q14	287	309	300	300
q15	q16	720	743	651	651
q17	1338	1343	1351	1343
q18	8094	7543	7384	7384
q19	1144	1109	1143	1109
q20	2216	2198	1920	1920
q21	5360	4624	4690	4624
q22	520	474	395	395
Total cold run time: 56608 ms
Total hot run time: 51377 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
TPC-DS: Total hot run time: 169357 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 1c7efc9ebb7ed4e2cc2c5c4c7ffc89d443be2e25, data reload: false

query5	4339	608	470	470
query6	463	212	194	194
query7	4852	614	348	348
query8	338	180	168	168
query9	8744	4020	4017	4017
query10	471	377	300	300
query11	5787	2214	1947	1947
query12	160	97	97	97
query13	1253	637	449	449
query14	6030	4668	4306	4306
query14_1	3737	3759	3753	3753
query15	197	204	172	172
query16	1003	469	427	427
query17	918	683	522	522
query18	2440	465	348	348
query19	207	188	144	144
query20	109	101	107	101
query21	237	158	132	132
query22	13031	13150	12828	12828
query23	17412	16359	16037	16037
query23_1	16095	16164	16011	16011
query24	7720	1715	1253	1253
query24_1	1233	1256	1267	1256
query25	548	447	385	385
query26	1334	353	221	221
query27	2595	578	399	399
query28	4505	2079	2040	2040
query29	1099	619	491	491
query30	342	261	233	233
query31	1126	1070	948	948
query32	114	63	65	63
query33	536	313	263	263
query34	1195	1100	640	640
query35	767	738	636	636
query36	769	777	695	695
query37	148	108	87	87
query38	1822	1644	1575	1575
query39	810	813	798	798
query39_1	778	791	780	780
query40	250	161	142	142
query41	67	64	65	64
query42	95	93	94	93
query43	316	314	274	274
query44	1442	775	762	762
query45	184	175	162	162
query46	1037	1192	720	720
query47	1538	1550	1432	1432
query48	398	413	282	282
query49	581	413	296	296
query50	1031	429	332	332
query51	10757	10435	10437	10435
query52	87	86	76	76
query53	263	284	192	192
query54	277	228	210	210
query55	74	71	66	66
query56	276	291	291	291
query57	1032	989	944	944
query58	280	285	256	256
query59	1536	1615	1366	1366
query60	319	285	256	256
query61	150	139	149	139
query62	399	317	269	269
query63	230	195	196	195
query64	2842	1012	825	825
query65	3878	3802	3837	3802
query66	1826	474	350	350
query67	28191	28211	28044	28044
query68	3116	1502	1026	1026
query69	404	301	260	260
query70	893	776	785	776
query71	345	337	309	309
query72	3000	2625	2453	2453
query73	835	777	461	461
query74	4657	4504	4284	4284
query75	2388	2323	1985	1985
query76	2348	1128	761	761
query77	348	377	281	281
query78	11256	11114	10557	10557
query79	1416	1108	739	739
query80	748	583	494	494
query81	465	323	290	290
query82	610	152	120	120
query83	390	325	314	314
query84	386	155	136	136
query85	943	646	513	513
query86	348	239	222	222
query87	1798	1795	1707	1707
query88	3774	2830	2820	2820
query89	398	323	294	294
query90	1839	196	189	189
query91	207	188	167	167
query92	63	56	52	52
query93	1643	1566	1038	1038
query94	608	365	309	309
query95	811	570	491	491
query96	1058	769	339	339
query97	2483	2464	2355	2355
query98	197	203	196	196
query99	710	721	607	607
Total cold run time: 255095 ms
Total hot run time: 169357 ms

@hello-stephen

Copy link
Copy Markdown
Contributor
ClickBench: Total hot run time: 23.82 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 1c7efc9ebb7ed4e2cc2c5c4c7ffc89d443be2e25, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.04
query3	0.25	0.14	0.13
query4	1.60	0.14	0.14
query5	0.24	0.22	0.22
query6	1.15	0.82	0.78
query7	0.04	0.00	0.00
query8	0.06	0.04	0.04
query9	0.37	0.30	0.31
query10	0.54	0.56	0.54
query11	0.19	0.14	0.13
query12	0.18	0.15	0.14
query13	0.47	0.46	0.45
query14	1.00	0.98	0.99
query15	0.59	0.59	0.58
query16	0.32	0.31	0.32
query17	1.09	1.09	1.08
query18	0.22	0.18	0.20
query19	2.01	1.98	1.97
query20	0.01	0.01	0.01
query21	15.43	0.22	0.13
query22	4.77	0.06	0.06
query23	16.16	0.31	0.12
query24	3.03	0.42	0.32
query25	0.12	0.05	0.04
query26	0.76	0.20	0.14
query27	0.04	0.04	0.03
query28	3.48	0.79	0.35
query29	12.49	4.09	3.21
query30	0.28	0.15	0.16
query31	2.77	0.55	0.32
query32	3.22	0.58	0.48
query33	3.19	3.24	3.19
query34	15.69	3.93	3.29
query35	3.22	3.26	3.19
query36	0.56	0.43	0.41
query37	0.09	0.06	0.06
query38	0.06	0.04	0.04
query39	0.04	0.03	0.03
query40	0.18	0.16	0.14
query41	0.08	0.03	0.03
query42	0.03	0.04	0.03
query43	0.04	0.03	0.04
Total cold run time: 96.17 s
Total hot run time: 23.82 s

@hello-stephen

Copy link
Copy Markdown
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (3/3) 🎉
Increment coverage report
Complete coverage report

@yujun777

yujun777 commented Aug 4, 2026

Copy link
Copy Markdown
Contributor Author

@yujun777 yujun777 closed this Aug 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants