parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths by pchintar · Pull Request #9797 · apache/arrow-rs

pchintar · 2026-04-23T07:47:54Z

Which issue does this PR close?

Closes DeltaByteArrayDecoder panics on invalid prefix lengths #9796 .

Rationale for this change

Currently, DeltaByteArrayDecoder::get assumes prefix lengths are always valid and directly slices previous_value. Invalid prefix lengths (negative or exceeding previous value length) can cause a panic instead of returning an error.

What changes are included in this PR?

Add validation for decoded prefix lengths:
- reject negative values
- reject values exceeding previous_value.len()
Return Err instead of panicking on invalid input
Add a regression test using corrupted encoded data

Are these changes tested?

Yes.

Added test_delta_byte_array_invalid_prefix_len_returns_error
Test:
- encodes valid data
- corrupts prefix-length stream
- verifies decoder returns Err (previously panicked)
All the other existing tests pass

Are there any user-facing changes?

No.

No API changes
Only improves error handling for invalid input

etseidl

Thanks @pchintar, this seems like a sensible fix. One nit and I think this needs a bit more test coverage.

etseidl · 2026-04-23T15:51:05Z

+                    let prefix_len_i32 = self.prefix_lengths[self.current_idx];
+                    if prefix_len_i32 < 0 {
+                        return Err(general_err!(
+                            "Invalid DELTA_BYTE_ARRAY prefix length {}",
+                            prefix_len_i32
+                        ));
+                    }
+                    let prefix_len = prefix_len_i32 as usize;


Suggested change

let prefix_len_i32 = self.prefix_lengths[self.current_idx];

if prefix_len_i32 < 0 {

return Err(general_err!(

"Invalid DELTA_BYTE_ARRAY prefix length {}",

prefix_len_i32

));

}

let prefix_len = prefix_len_i32 as usize;

let prefix_len = usize::try_from(self.prefix_lengths[self.current_idx])?;

A little less verbose. 😄

I notice the added test doesn't cover this possibility.

Thanks @etseidl for the suggestion — updated to use usize::try_from(...) with proper error mapping.
Also added an additional test test_delta_byte_array_negative_prefix_len_returns_error to cover the negative prefix length case so that path is now exercised as well.

etseidl · 2026-04-23T17:45:05Z

run benchmark parquet_round_trip
env:
BENCH_FILTER:read.*delta_byte

adriangbot · 2026-04-23T17:45:07Z

Hi @etseidl, your benchmark configuration could not be parsed (#9797 (comment)).

Error: invalid configuration: env: invalid type: string "BENCH_FILTER:read.*delta_byte", expected a map at line 2 column 3

Supported benchmarks:

Standard: (none)
Criterion: (any)

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Per-side configuration (run benchmark tpch followed by):

env:
SHARED_SETTING: enabled
baseline:
ref: v45.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

File an issue against this benchmark runner

etseidl · 2026-04-23T17:55:34Z

run benchmark parquet_round_trip

env:
  BENCH_FILTER:read.*delta_byte

adriangbot · 2026-04-23T17:55:35Z

Hi @etseidl, your benchmark configuration could not be parsed (#9797 (comment)).

Error: invalid configuration: env: invalid type: string "BENCH_FILTER:read.*delta_byte", expected a map at line 2 column 3

Supported benchmarks:

Standard: (none)
Criterion: (any)

Usage:

run benchmark <name>           # run specific benchmark(s)
run benchmarks                 # run default suite
run benchmarks <name1> <name2> # run specific benchmarks

Per-side configuration (run benchmark tpch followed by):

env:
SHARED_SETTING: enabled
baseline:
ref: v45.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 1G
changed:
ref: v46.0.0
env:
DATAFUSION_RUNTIME_MEMORY_LIMIT: 2G

File an issue against this benchmark runner

etseidl · 2026-04-23T17:57:24Z

run benchmark parquet_round_trip

env:
  BENCH_FILTER: read.*delta_byte

adriangbot · 2026-04-23T18:01:09Z

🤖 Arrow criterion benchmark running (GKE) | trigger
Instance: c4a-highmem-16 (12 vCPU / 65 GiB) | Linux bench-c4306626542-1782-sf5xd 6.12.55+ #1 SMP Sun Feb 1 08:59:41 UTC 2026 aarch64 GNU/Linux

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Comparing delta_byte_array (542824b) to b93240a (merge-base) diff
BENCH_NAME=parquet_round_trip
BENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench parquet_round_trip
BENCH_FILTER=read.*delta_byte
Results will be posted here when complete

File an issue against this benchmark runner

adriangbot · 2026-04-23T18:04:39Z

🤖 Arrow criterion benchmark completed (GKE) | trigger

Instance: c4a-highmem-16 (12 vCPU / 65 GiB)

CPU Details (lscpu)

Architecture:                            aarch64
CPU op-mode(s):                          64-bit
Byte Order:                              Little Endian
CPU(s):                                  16
On-line CPU(s) list:                     0-15
Vendor ID:                               ARM
Model name:                              Neoverse-V2
Model:                                   1
Thread(s) per core:                      1
Core(s) per cluster:                     16
Socket(s):                               -
Cluster(s):                              1
Stepping:                                r0p1
BogoMIPS:                                2000.00
Flags:                                   fp asimd evtstrm aes pmull sha1 sha2 crc32 atomics fphp asimdhp cpuid asimdrdm jscvt fcma lrcpc dcpop sha3 sm3 sm4 asimddp sha512 sve asimdfhm dit uscat ilrcpc flagm sb paca pacg dcpodp sve2 sveaes svepmull svebitperm svesha3 svesm4 flagm2 frint svei8mm svebf16 i8mm bf16 dgh rng bti
L1d cache:                               1 MiB (16 instances)
L1i cache:                               1 MiB (16 instances)
L2 cache:                                32 MiB (16 instances)
L3 cache:                                80 MiB (1 instance)
NUMA node(s):                            1
NUMA node0 CPU(s):                       0-15
Vulnerability Gather data sampling:      Not affected
Vulnerability Indirect target selection: Not affected
Vulnerability Itlb multihit:             Not affected
Vulnerability L1tf:                      Not affected
Vulnerability Mds:                       Not affected
Vulnerability Meltdown:                  Not affected
Vulnerability Mmio stale data:           Not affected
Vulnerability Reg file data sampling:    Not affected
Vulnerability Retbleed:                  Not affected
Vulnerability Spec rstack overflow:      Not affected
Vulnerability Spec store bypass:         Mitigation; Speculative Store Bypass disabled via prctl
Vulnerability Spectre v1:                Mitigation; __user pointer sanitization
Vulnerability Spectre v2:                Mitigation; CSV2, BHB
Vulnerability Srbds:                     Not affected
Vulnerability Tsa:                       Not affected
Vulnerability Tsx async abort:           Not affected
Vulnerability Vmscape:                   Not affected

Details

group                                    delta_byte_array                       main
-----                                    ----------------                       ----
read Binary(100) delta_byte_array        1.00      8.4±0.14ms        ? ?/sec    1.08      9.1±0.23ms        ? ?/sec
read Binary(20) delta_byte_array         1.01      4.9±0.04ms        ? ?/sec    1.00      4.9±0.04ms        ? ?/sec
read Fixed(16) delta_byte_array          1.00      3.4±0.06ms        ? ?/sec    1.01      3.5±0.11ms        ? ?/sec
read Fixed(2) delta_byte_array           1.00      3.2±0.01ms        ? ?/sec    1.00      3.2±0.01ms        ? ?/sec
read String(100) delta_byte_array        1.00      9.3±0.32ms        ? ?/sec    1.00      9.3±0.16ms        ? ?/sec
read String(20) delta_byte_array         1.01      5.2±0.03ms        ? ?/sec    1.00      5.2±0.02ms        ? ?/sec
read StringView(100) delta_byte_array    1.00      8.6±0.49ms        ? ?/sec    1.07      9.3±0.27ms        ? ?/sec
read StringView(20) delta_byte_array     1.00      4.9±0.04ms        ? ?/sec    1.00      4.9±0.05ms        ? ?/sec

Resource Usage

base (merge-base)

Metric	Value
Wall time	85.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	82.0s
CPU sys	0.9s
Peak spill	0 B

branch

Metric	Value
Wall time	85.0s
Peak memory	4.3 GiB
Avg memory	4.2 GiB
CPU user	81.0s
CPU sys	0.2s
Peak spill	0 B

File an issue against this benchmark runner

etseidl

There doesn't appear to be any impact on read performance. Looks good to me. Thanks @pchintar

pchintar · 2026-04-23T18:39:01Z

Thank you @etseidl

…h test

github-actions Bot added the parquet Changes to the parquet crate label Apr 23, 2026

pchintar force-pushed the delta_byte_array branch from c463a0a to e0201f5 Compare April 23, 2026 09:10

etseidl reviewed Apr 23, 2026

View reviewed changes

etseidl approved these changes Apr 23, 2026

View reviewed changes

pchintar added 3 commits April 23, 2026 18:48

final format fix

5985077

Apply rustfmt formatting

19f860d

parquet: use try_from for prefix length and add negative prefix lengt…

69fcdef

…h test

pchintar force-pushed the delta_byte_array branch from 542824b to 69fcdef Compare April 23, 2026 22:49

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths#9797

parquet: fix panic in DeltaByteArrayDecoder on invalid prefix lengths#9797
pchintar wants to merge 3 commits intoapache:mainfrom
pchintar:delta_byte_array

pchintar commented Apr 23, 2026

Uh oh!

etseidl left a comment

Uh oh!

etseidl Apr 23, 2026

Uh oh!

pchintar Apr 23, 2026 •

edited

Loading

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl left a comment

Uh oh!

pchintar commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

pchintar commented Apr 23, 2026

Which issue does this PR close?

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

etseidl Apr 23, 2026

Choose a reason for hiding this comment

Uh oh!

pchintar Apr 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

adriangbot commented Apr 23, 2026

Uh oh!

etseidl left a comment

Choose a reason for hiding this comment

Uh oh!

pchintar commented Apr 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

pchintar Apr 23, 2026 •

edited

Loading