[fix](parquet) Handle empty value sections by acarofpigs · Pull Request #66556 · apache/doris

acarofpigs · 2026-08-06T14:44:05Z

• ### What problem does this PR solve?

Issue Number: close #66430

Related PR: None

Problem Summary:
A compressed Parquet DataPageV2 can legally contain definition levels
but no physical values when every nullable value is NULL. The legacy
reader passed the resulting null, zero-length slice to the BOOLEAN
PLAIN decoder and aborted the BE in BatchedBitReader::Reset().

Use a dedicated empty-value-section decoder for these pages. Accept the
page when definition levels require no physical values, and return
Corruption when values are required but absent. Update page progress
only after decoding succeeds so failures leave reader state unchanged.

Release note

Fix a BE crash when reading Parquet pages with empty value sections.

Check List (For Author)

Test
- Regression test
- Unit Test
- Manual test (add detailed scripts or steps below)
  - ./run-be-ut.sh --run --filter=ParquetColumnChunkReaderTest.* -j 2
  - All 11 tests passed with ASAN.
  - ./build.sh --be -j 2
  - BE build succeeded.
- No need to test or manual test. Explain why:
  - This is a refactor/code format and no logic has been changed.
  - Previous test can cover this change.
  - No code files have been changed.
  - Other reason
Behavior changed:
- No.
- Yes. Valid all-NULL Parquet pages are accepted. Pages whose
  definition levels require missing physical values return
  Corruption instead of aborting the BE.
Does this need documentation?
- No.
- Yes.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

hello-stephen · 2026-08-06T14:44:11Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

acarofpigs · 2026-08-06T14:51:36Z

run buildall

hello-stephen · 2026-08-06T15:35:38Z

TPC-H: Total hot run time: 28635 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit fd0d0f1451a03d12dde625eb84d09101206b812f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17730	3921	3915	3915
q2	1987	322	203	203
q3	10300	1405	780	780
q4	4681	461	337	337
q5	7476	822	547	547
q6	179	164	132	132
q7	712	789	586	586
q8	9335	1504	1545	1504
q9	5754	4060	4006	4006
q10	6746	1624	1340	1340
q11	514	348	324	324
q12	759	578	444	444
q13	18112	3247	2713	2713
q14	254	257	238	238
q15	q16	727	723	663	663
q17	950	920	991	920
q18	6615	5690	5601	5601
q19	1394	1324	1104	1104
q20	818	714	632	632
q21	5918	2555	2349	2349
q22	422	356	297	297
Total cold run time: 101383 ms
Total hot run time: 28635 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4246	4167	4179	4167
q2	269	312	210	210
q3	4533	4852	4367	4367
q4	2120	2229	1383	1383
q5	4166	4088	4096	4088
q6	229	174	123	123
q7	1661	1573	1391	1391
q8	2495	2143	2035	2035
q9	7217	7125	7201	7125
q10	4276	4279	3863	3863
q11	535	410	368	368
q12	730	741	503	503
q13	3243	3528	3067	3067
q14	290	289	274	274
q15	q16	688	712	621	621
q17	1296	1306	1316	1306
q18	12119	11030	11767	11030
q19	1168	1180	1165	1165
q20	2221	2251	1975	1975
q21	5602	4821	4614	4614
q22	503	479	410	410
Total cold run time: 59607 ms
Total hot run time: 54085 ms

hello-stephen · 2026-08-06T15:46:12Z

TPC-DS: Total hot run time: 164318 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit fd0d0f1451a03d12dde625eb84d09101206b812f, data reload: false

query5	4317	578	450	450
query6	461	216	192	192
query7	4900	616	348	348
query8	321	156	142	142
query9	8790	3975	3921	3921
query10	480	352	296	296
query11	5786	2185	2005	2005
query12	148	105	93	93
query13	1239	631	398	398
query14	6054	4249	3972	3972
query14_1	3770	3753	3724	3724
query15	209	196	182	182
query16	1004	489	400	400
query17	936	687	555	555
query18	2434	476	342	342
query19	210	192	147	147
query20	103	101	102	101
query21	233	151	136	136
query22	12921	13070	12770	12770
query23	15694	14892	14534	14534
query23_1	14674	14682	14632	14632
query24	7671	1693	1228	1228
query24_1	1250	1222	1230	1222
query25	561	452	391	391
query26	1308	341	219	219
query27	2609	596	373	373
query28	4494	2022	1986	1986
query29	1028	595	454	454
query30	334	264	220	220
query31	1164	1111	1045	1045
query32	106	61	57	57
query33	510	292	225	225
query34	1161	1110	638	638
query35	722	731	626	626
query36	763	758	687	687
query37	152	100	84	84
query38	1824	1778	1684	1684
query39	844	823	777	777
query39_1	786	788	774	774
query40	260	162	145	145
query41	67	63	61	61
query42	91	88	89	88
query43	312	318	269	269
query44	1413	767	754	754
query45	182	171	170	170
query46	1057	1177	717	717
query47	1511	1551	1439	1439
query48	423	421	285	285
query49	583	412	291	291
query50	1047	424	338	338
query51	10682	10325	10544	10325
query52	83	84	75	75
query53	271	284	193	193
query54	271	222	217	217
query55	72	70	64	64
query56	307	280	300	280
query57	1040	998	944	944
query58	276	245	248	245
query59	1505	1585	1376	1376
query60	307	271	255	255
query61	147	146	147	146
query62	399	316	261	261
query63	227	197	187	187
query64	2817	1019	812	812
query65	3903	3774	3804	3774
query66	1821	471	356	356
query67	28067	27381	27909	27381
query68	3224	1611	997	997
query69	414	287	254	254
query70	883	784	786	784
query71	345	349	317	317
query72	3134	2803	2455	2455
query73	825	749	463	463
query74	4618	4491	4272	4272
query75	2386	2336	2002	2002
query76	2352	1152	743	743
query77	343	409	266	266
query78	11188	11107	10484	10484
query79	1435	1141	740	740
query80	1242	548	449	449
query81	516	324	279	279
query82	623	168	136	136
query83	382	318	293	293
query84	328	160	129	129
query85	937	583	530	530
query86	404	233	230	230
query87	1972	1943	1805	1805
query88	3710	2766	2738	2738
query89	380	313	279	279
query90	1927	192	186	186
query91	199	189	160	160
query92	61	56	54	54
query93	1714	1506	929	929
query94	691	373	309	309
query95	781	496	466	466
query96	1029	841	353	353
query97	2492	2441	2343	2343
query98	195	186	180	180
query99	746	740	610	610
Total cold run time: 252854 ms
Total hot run time: 164318 ms

hello-stephen · 2026-08-06T15:50:57Z

ClickBench: Total hot run time: 23.69 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit fd0d0f1451a03d12dde625eb84d09101206b812f, data reload: false

query1	0.01	0.01	0.00
query2	0.09	0.04	0.04
query3	0.25	0.13	0.13
query4	1.61	0.14	0.13
query5	0.23	0.22	0.21
query6	1.16	0.82	0.81
query7	0.04	0.01	0.00
query8	0.06	0.04	0.04
query9	0.37	0.33	0.31
query10	0.58	0.54	0.55
query11	0.18	0.13	0.13
query12	0.18	0.14	0.14
query13	0.46	0.46	0.47
query14	1.00	0.98	0.98
query15	0.60	0.61	0.58
query16	0.31	0.32	0.33
query17	1.07	1.05	1.09
query18	0.21	0.19	0.19
query19	2.09	1.91	1.98
query20	0.02	0.02	0.01
query21	15.44	0.19	0.14
query22	5.03	0.06	0.06
query23	16.11	0.32	0.12
query24	2.99	0.41	0.33
query25	0.12	0.04	0.04
query26	0.74	0.21	0.14
query27	0.05	0.04	0.04
query28	3.56	0.75	0.33
query29	12.48	4.03	3.17
query30	0.28	0.15	0.15
query31	2.77	0.54	0.32
query32	3.22	0.58	0.48
query33	3.14	3.16	3.16
query34	15.71	3.88	3.24
query35	3.24	3.21	3.23
query36	0.55	0.43	0.42
query37	0.08	0.06	0.06
query38	0.05	0.04	0.04
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.02
query43	0.04	0.04	0.03
Total cold run time: 96.46 s
Total hot run time: 23.69 s

hello-stephen · 2026-08-06T18:51:14Z

BE Regression && UT Coverage Report

Increment line coverage 67.35% (33/49) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.93% (32304/42543)
Line Coverage	60.81% (362286/595784)
Region Coverage	57.49% (304917/530357)
Branch Coverage	58.94% (137818/233808)

### What problem does this PR solve? Issue Number: close apache#66430 Related PR: None Problem Summary: A compressed Parquet DataPageV2 can legally contain definition levels but no physical values when every nullable value is NULL. The legacy reader passed the resulting null, zero-length slice to the BOOLEAN PLAIN decoder and aborted the BE in BatchedBitReader::Reset(). Use a dedicated empty-value-section decoder for these pages. Accept the page when definition levels require no physical values, and return Corruption when values are required but absent. Update page progress only after decoding succeeds so failures leave reader state unchanged. ### Release note Fix a BE crash when reading Parquet pages with empty value sections. ### Check List (For Author) - Test  - [ ] Regression test - [x] Unit Test - ./run-be-ut.sh --run --filter=ParquetColumnChunkReaderTest.* -j 2 - All 11 tests passed with ASAN. - [x] Manual test (add detailed scripts or steps below) - ./build.sh --be -j 2 - BE build succeeded. - [ ] No need to test or manual test. Explain why: - [ ] This is a refactor/code format and no logic has been changed. - [ ] Previous test can cover this change. - [ ] No code files have been changed. - [ ] Other reason  - Behavior changed: - [ ] No. - [x] Yes. Valid all-NULL Parquet pages are accepted. Pages whose definition levels require missing physical values return Corruption instead of aborting the BE. - Does this need documentation? - [x] No. - [ ] Yes.  ### Check List (For Reviewer who merge this PR) - [ ] Confirm the release note - [ ] Confirm test cases - [ ] Confirm document - [ ] Add branch pick label

acarofpigs · 2026-08-06T22:59:36Z

run buildall

hello-stephen · 2026-08-07T00:01:18Z

TPC-H: Total hot run time: 28412 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit aba31ae451b74c82a2a7796bc6e941e740e59d8e, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17899	3936	3937	3936
q2	1990	310	193	193
q3	10310	1411	783	783
q4	4683	461	338	338
q5	7499	836	528	528
q6	183	171	143	143
q7	759	772	576	576
q8	9836	1571	1504	1504
q9	5739	4030	4000	4000
q10	6780	1617	1353	1353
q11	511	340	314	314
q12	752	578	447	447
q13	18095	3164	2658	2658
q14	261	256	239	239
q15	q16	734	726	661	661
q17	967	968	1007	968
q18	6519	5596	5526	5526
q19	1303	1256	1052	1052
q20	763	680	608	608
q21	5827	2559	2287	2287
q22	412	351	298	298
Total cold run time: 101822 ms
Total hot run time: 28412 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4258	4188	4161	4161
q2	269	311	205	205
q3	4527	4864	4343	4343
q4	2158	2229	1414	1414
q5	4198	4035	4073	4035
q6	220	171	127	127
q7	1662	1607	1412	1412
q8	2559	2161	2030	2030
q9	7176	7181	7309	7181
q10	4288	4269	3844	3844
q11	549	416	358	358
q12	709	745	514	514
q13	3087	3405	3048	3048
q14	302	307	282	282
q15	q16	732	699	627	627
q17	1295	1301	1281	1281
q18	12130	11007	11745	11007
q19	1208	1167	1132	1132
q20	2196	2210	1955	1955
q21	5650	4717	4749	4717
q22	494	444	426	426
Total cold run time: 59667 ms
Total hot run time: 54099 ms

hello-stephen · 2026-08-07T00:11:53Z

TPC-DS: Total hot run time: 166157 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit aba31ae451b74c82a2a7796bc6e941e740e59d8e, data reload: false

query5	4306	580	458	458
query6	464	219	192	192
query7	4875	570	341	341
query8	318	153	141	141
query9	8773	3986	3986	3986
query10	446	355	293	293
query11	5727	2166	1973	1973
query12	175	98	93	93
query13	1253	586	425	425
query14	6029	4228	3953	3953
query14_1	3730	3742	3727	3727
query15	194	194	172	172
query16	978	448	444	444
query17	911	670	527	527
query18	2408	461	329	329
query19	200	186	150	150
query20	104	102	111	102
query21	230	157	133	133
query22	13120	12954	12792	12792
query23	15539	15088	14607	14607
query23_1	14748	14663	14644	14644
query24	7530	1705	1221	1221
query24_1	1236	1252	1211	1211
query25	513	428	342	342
query26	1321	371	214	214
query27	2585	593	370	370
query28	4550	2019	2020	2019
query29	1050	601	460	460
query30	339	261	224	224
query31	1173	1112	1043	1043
query32	113	62	58	58
query33	506	296	232	232
query34	1170	1140	659	659
query35	722	739	644	644
query36	767	768	704	704
query37	156	106	91	91
query38	1839	1775	1663	1663
query39	828	846	785	785
query39_1	782	786	790	786
query40	243	163	144	144
query41	88	63	62	62
query42	93	95	93	93
query43	319	332	277	277
query44	1451	765	746	746
query45	185	173	167	167
query46	1044	1226	715	715
query47	1565	1514	1407	1407
query48	402	444	311	311
query49	577	392	286	286
query50	1055	431	334	334
query51	10375	10629	10753	10629
query52	93	87	75	75
query53	256	288	200	200
query54	272	233	237	233
query55	78	70	66	66
query56	316	293	303	293
query57	1024	1013	963	963
query58	287	257	244	244
query59	1584	1643	1416	1416
query60	305	271	261	261
query61	148	148	145	145
query62	409	321	267	267
query63	255	200	202	200
query64	2910	1197	1002	1002
query65	3902	3888	3862	3862
query66	1850	471	361	361
query67	28142	28075	27972	27972
query68	3317	1653	1051	1051
query69	394	282	255	255
query70	889	788	778	778
query71	371	325	325	325
query72	2961	2619	2314	2314
query73	848	803	414	414
query74	4554	4476	4265	4265
query75	2350	2318	1979	1979
query76	2342	1165	775	775
query77	346	404	263	263
query78	11128	11087	10546	10546
query79	1424	1187	796	796
query80	659	533	467	467
query81	461	333	288	288
query82	617	173	139	139
query83	412	330	304	304
query84	328	164	133	133
query85	918	607	499	499
query86	324	248	225	225
query87	1985	1979	1830	1830
query88	3757	2855	2791	2791
query89	393	342	291	291
query90	1858	196	190	190
query91	200	194	160	160
query92	65	60	53	53
query93	1574	1550	978	978
query94	532	338	315	315
query95	791	505	582	505
query96	1061	808	365	365
query97	2477	2443	2350	2350
query98	195	186	186	186
query99	744	728	616	616
Total cold run time: 251384 ms
Total hot run time: 166157 ms

hello-stephen · 2026-08-07T00:16:39Z

ClickBench: Total hot run time: 23.77 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit aba31ae451b74c82a2a7796bc6e941e740e59d8e, data reload: false

query1	0.00	0.00	0.01
query2	0.09	0.04	0.05
query3	0.25	0.14	0.13
query4	1.62	0.13	0.14
query5	0.23	0.22	0.21
query6	1.15	0.81	0.81
query7	0.04	0.01	0.00
query8	0.05	0.03	0.04
query9	0.37	0.30	0.30
query10	0.59	0.55	0.54
query11	0.20	0.13	0.14
query12	0.18	0.14	0.13
query13	0.47	0.47	0.47
query14	1.00	1.00	0.98
query15	0.61	0.57	0.62
query16	0.31	0.31	0.31
query17	1.08	1.09	1.09
query18	0.20	0.21	0.18
query19	1.95	1.99	1.89
query20	0.02	0.01	0.01
query21	15.41	0.21	0.15
query22	4.86	0.05	0.05
query23	16.14	0.31	0.12
query24	2.93	0.42	0.31
query25	0.11	0.05	0.03
query26	0.72	0.22	0.15
query27	0.04	0.04	0.03
query28	3.56	0.78	0.33
query29	12.46	4.10	3.25
query30	0.28	0.16	0.15
query31	2.77	0.56	0.32
query32	3.21	0.58	0.48
query33	3.16	3.18	3.23
query34	15.66	3.93	3.27
query35	3.22	3.22	3.22
query36	0.56	0.42	0.42
query37	0.09	0.06	0.07
query38	0.04	0.04	0.03
query39	0.04	0.03	0.03
query40	0.17	0.16	0.15
query41	0.09	0.03	0.03
query42	0.04	0.03	0.03
query43	0.04	0.04	0.03
Total cold run time: 96.01 s
Total hot run time: 23.77 s

hello-stephen · 2026-08-07T02:22:02Z

BE Regression && UT Coverage Report

Increment line coverage 100.00% (38/38) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.97% (32320/42543)
Line Coverage	60.81% (362305/595773)
Region Coverage	57.41% (304478/530347)
Branch Coverage	58.93% (137776/233796)

acarofpigs force-pushed the fix/66430-empty-value-section-crash branch from 23de867 to fd0d0f1 Compare August 6, 2026 14:44

acarofpigs force-pushed the fix/66430-empty-value-section-crash branch from fd0d0f1 to aba31ae Compare August 6, 2026 22:53

acarofpigs mentioned this pull request Aug 7, 2026

[Bug] BE Crash with SIGABRT on Load queries and throws bit_stream_utils.h:187] Check failed: buffer != nullptr #66430

Open

3 tasks

morningman added the dev/4.1.x label Aug 10, 2026

Navigation Menu

Uh oh!

[fix](parquet) Handle empty value sections - #66556

[fix](parquet) Handle empty value sections#66556
acarofpigs wants to merge 1 commit into
apache:masterfrom
acarofpigs:fix/66430-empty-value-section-crash

acarofpigs commented Aug 6, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

acarofpigs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

acarofpigs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

acarofpigs commented Aug 6, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Release note

Check List (For Author)

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

acarofpigs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 6, 2026

BE Regression && UT Coverage Report

Uh oh!

acarofpigs commented Aug 6, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

Uh oh!

hello-stephen commented Aug 7, 2026

BE Regression && UT Coverage Report

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

acarofpigs commented Aug 6, 2026 •

edited

Loading