Fuzzing Crash: VortexError in file_io

[github-actions]

Fuzzing Crash Report

Analysis

Crash Location: vortex-array/src/arrays/dict/array.rs:167:validate_all_values_referenced

Error Message:

validation should succeed when all values are referenced:
  Other error: value in dict not referenced

Stack Trace

stack backtrace:
   0: __rustc::rust_begin_unwind
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/std/src/panicking.rs:689:5
   1: core::panicking::panic_fmt
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/panicking.rs:80:14
   2: panic_display<vortex_error::VortexError>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/panicking.rs:259:5
   3: {closure#1}<(), vortex_error::VortexError>
             at ./vortex-error/src/lib.rs:500:9
   4: unwrap_or_else<(), vortex_error::VortexError, vortex_error::{impl#11}::vortex_expect::{closure_env#1}<(), vortex_error::VortexError>>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/result.rs:1622:23
   5: vortex_expect<(), vortex_error::VortexError>
             at ./vortex-error/src/lib.rs:340:14
   6: set_all_values_referenced
             at ./vortex-array/src/arrays/dict/array.rs:90:18
   7: compress
             at ./vortex-btrblocks/src/compressor/float/mod.rs:453:22
   8: estimate_compression_ratio_with_sampling<vortex_btrblocks::compressor::float::DictScheme>
             at ./vortex-btrblocks/src/scheme.rs:121:14
   9: expected_compression_ratio
             at ./vortex-btrblocks/src/compressor/float/mod.rs:422:14
  10: choose_scheme<vortex_btrblocks::compressor::float::FloatCompressor>
             at ./vortex-btrblocks/src/compressor/mod.rs:110:32
  11: compress<vortex_btrblocks::compressor::float::FloatCompressor>
             at ./vortex-btrblocks/src/compressor/mod.rs:165:32
  12: compress_canonical
             at ./vortex-btrblocks/src/canonical_compressor.rs:225:26
  13: compress
             at ./vortex-btrblocks/src/canonical_compressor.rs:126:14
  14: compress_chunk
             at ./vortex-layout/src/layouts/compressed.rs:35:23
  15: {closure#0}
             at ./vortex-layout/src/layouts/compressed.rs:124:49
  16: {closure#0}<vortex_layout::layouts::compressed::{impl#5}::write_stream::{async_block#0}::{closure#0}::{closure_env#0}, core::result::Result<(vortex_layout::sequence::SequenceId, alloc::sync::Arc<dyn vortex_array::array::DynArray, alloc::alloc::Global>), vortex_error::VortexError>>
             at ./vortex-io/src/runtime/handle.rs:115:32
  17: call_once<vortex_io::runtime::handle::{impl#0}::spawn_cpu::{closure_env#0}<vortex_layout::layouts::compressed::{impl#5}::write_stream::{async_block#0}::{closure#0}::{closure_env#0}, core::result::Result<(vortex_layout::sequence::SequenceId, alloc::sync::Arc<dyn vortex_array::array::DynArray, alloc::alloc::Global>), vortex_error::VortexError>>, ()>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/core/src/ops/function.rs:250:5
  18: call_once<(), (dyn core::ops::function::FnOnce<(), Output=()> + core::marker::Send), alloc::alloc::Global>
             at /rustc/db3e99bbab28c6ca778b13222becdea54533d908/library/alloc/src/boxed.rs:2206:9
  19: {async_block#0}
   ... (66 more frames truncated)

Root Cause Analysis

The crash is a panic in DictArray::set_all_values_referenced (vortex-array/src/arrays/dict/array.rs:167) where a debug assertion validates that all dictionary values are referenced by codes, but finds unreferenced values. This occurs during float dictionary compression in vortex-btrblocks when the compressor compresses the codes and values separately, then reassembles them with set_all_values_referenced(true), but the compression of codes can alter which values are actually referenced (e.g., by deduplicating or reindexing codes without pruning the corresponding values). The fix should either prune unreferenced values from the dictionary after compression, or avoid propagating the all_values_referenced flag when codes have been recompressed.

Summary

• Target: file_io
• Crash File: crash-df5d192e1ad041c443245ff29a78d9af6e6efae7
• Branch: develop
• Commit: 18359a8
• Crash Artifact: https://github.com/vortex-data/vortex/actions/runs/23369138954/artifacts/6036301549

Reproduce

cargo +nightly fuzz run -D --sanitizer=none file_io ./fuzz/artifacts/file_io/crash-df5d192e1ad041c443245ff29a78d9af6e6efae7 -- -rss_limit_mb=0

Reproduction Steps

1. Download the crash artifact: https://github.com/vortex-data/vortex/actions/runs/23369138954/artifacts/6036301549

2. Assuming you download the zipfile to ~/Downloads, and your working directory is the repository root:

# Create the artifacts directory if you haven't already.
mkdir -p ./fuzz/artifacts

# Move the zipfile.
mv ~/Downloads/file_io-crash-artifacts.zip ./fuzz/artifacts/

# Unzip the zipfile.
unzip ./fuzz/artifacts/file_io-crash-artifacts.zip -d ./fuzz/artifacts/

# You can remove the zipfile now if you want to.
rm ./fuzz/artifacts/file_io-crash-artifacts.zip

3. Reproduce the crash:

cargo +nightly fuzz run -D --sanitizer=none file_io ./fuzz/artifacts/file_io/crash-df5d192e1ad041c443245ff29a78d9af6e6efae7 -- -rss_limit_mb=0

If you want a backtrace:

RUST_BACKTRACE=1 cargo +nightly fuzz run -D --sanitizer=none file_io ./fuzz/artifacts/file_io/crash-df5d192e1ad041c443245ff29a78d9af6e6efae7 -- -rss_limit_mb=0

RUST_BACKTRACE=full cargo +nightly fuzz run -D --sanitizer=none file_io ./fuzz/artifacts/file_io/crash-df5d192e1ad041c443245ff29a78d9af6e6efae7 -- -rss_limit_mb=0

Single command to get a backtrace

mkdir -p ./fuzz/artifacts
mv ~/Downloads/file_io-crash-artifacts.zip ./fuzz/artifacts/
unzip ./fuzz/artifacts/file_io-crash-artifacts.zip -d ./fuzz/artifacts/
rm ./fuzz/artifacts/file_io-crash-artifacts.zip
RUST_BACKTRACE=1 cargo +nightly fuzz run -D --sanitizer=none file_io ./fuzz/artifacts/file_io/crash-df5d192e1ad041c443245ff29a78d9af6e6efae7 -- -rss_limit_mb=0

---

Auto-created by fuzzing workflow

[github-actions]

Seen 4 times