perf: reduce byte array materialization during Arrow export

[ClSlaid]

Important

AI-assisted development disclosure

GPT-5.6 Sol assisted with this PR. A human reviewed the changes.

Validation ran 413 tests. The benchmark suite covers 744 planned scenarios. This value is scenario coverage. It is not source line coverage.

Benchmark results

The end-to-end benchmark reads one Utf8/FSST Vortex file. It exports the same input to Arrow BinaryArray and StringArray.

Each value is the median wall time for one run.

Run	BinaryArray old	BinaryArray new	Change	StringArray old	StringArray new	Change
1	31.73 ms	24.30 ms	23.4% faster	31.82 ms	24.49 ms	23.0% faster
2	31.88 ms	21.15 ms	33.7% faster	31.83 ms	21.26 ms	33.2% faster
3	28.33 ms	22.30 ms	21.3% faster	28.32 ms	22.51 ms	20.5% faster
Median	31.73 ms	22.30 ms	29.7% faster	31.82 ms	22.51 ms	29.3% faster

The new matcher avoids an intermediate canonical VarBinView for both Arrow output types.

Representative direct Offset export results:

Encoding	Old matcher	New matcher	Change
FSST, non-null	1.193 ms	708.5 µs	40.6% faster
OnPair, non-null	1.008 ms	536.3 µs	46.8% faster
Zstd, non-null	2.168 ms	1.551 ms	28.5% faster
FSST, nullable	1.048 ms	505.1 µs	51.8% faster
OnPair, nullable	937.9 µs	380.4 µs	59.4% faster

read_all has no clear improvement. It returns a top-level ChunkedArray. The old matcher already used the specialized Chunked append path.

Purpose

Arrow OffsetArray export can create an intermediate canonical VarBinView. Some encodings can append directly to the Arrow offset builder. This PR uses the direct path when it is faster.

Changes

• Stop execution at encodings that can append directly to an offset builder.
• Continue execution through Dict, Filter, ScalarFn, and Slice.
• Add string export benchmarks for single and nested operators to vortex-arrow.
• Add an end-to-end file scan benchmark to the vortex crate.
• Keep vortex-file free of Arrow test dependencies.

Scenario coverage

The string benchmark defines 180 input scenarios:

• 5 encodings: Offset, View, FSST, OnPair, and Zstd.
• 3 structures: Flat, Dict, and Chunked.
• 6 operators: Identity, Filter, Take, Slice, Mask, and Zip.
• 2 validity modes: non-null and nullable.

The benchmark runs these scenarios through three output paths:

• 360 Arrow export scenarios: 180 inputs and 2 Arrow layouts.
• 180 direct Offset builder scenarios.
• 180 direct View builder scenarios.

A focused suite adds 24 nested operator scenarios for Arrow Offset export:

• 3 encodings: FSST, OnPair, and Zstd.
• 4 ordered operator pairs: Take-Filter, Filter-Take, Slice-Filter, and Filter-Slice.
• 2 validity modes: non-null and nullable.

The total is 744 planned scenarios. The file benchmark adds four scenarios. It tests two Arrow output types and two scan collection modes.

Benchmark method

The old and new binaries used the same benchmark source and Cargo profile.

The benchmark created one in-memory Vortex file. The file contained 1,048,576 Utf8 rows in 16 chunks. The default compressor stored each scan batch as FSST. Both output cases read this same file.

The setup created the file before timing started. The timed region performed these steps:

1. Scan the Vortex file.
2. Read one batch.
3. Extract the Utf8 field.
4. Export the field to Arrow StringArray or BinaryArray.

Each process used CPU 6. Each case used 80 samples. Divan used a minimum time of 2 seconds and a maximum time of 3 seconds. The run order was old/new, new/old, and old/new.

Command:

taskset -c 6 <benchmark-binary> \
  --bench file_to_offset_batches \
  --sample-count 80 \
  --min-time 2 \
  --max-time 3

The direct Offset microbenchmarks ran on the same machine with the standard scheduler.

Known limitation

Some synthetic Zip scenarios are slower. Execution continues through ScalarFn and stops at its result encoding. FSST Chunked Zip increased from 2.063 ms to 5.519 ms in the largest measured case. The end-to-end file benchmark does not use this lazy Zip path.

Benchmark environment

Hardware

Item	Value
CPU	AMD Ryzen 7 5800X3D
CPU topology	8 cores, 16 threads, 1 socket
Architecture	x86-64
L3 cache	96 MiB
Memory	62 GiB
CPU affinity	CPU 6 for the paired file benchmark
CPU governor	powersave
CPU boost	enabled

Software

Item	Value
Operating system	CachyOS
Kernel	Linux 7.1.6-1-cachyos
Rust	rustc 1.91.0, LLVM 21.1.2
Cargo	1.91.0
Arrow	58.4.0
Benchmark harness	codspeed-divan-compat 5.0.1
Cargo profile	bench, 16 codegen units, debug info enabled, LTO disabled

The machine ran a desktop session during the test. The test did not isolate the full system. CPU affinity limited each measured process to one logical CPU.

Validation

• cargo nextest run -p vortex-arrow -p vortex-file -p vortex — 413 tests passed.
• cargo clippy -p vortex-arrow -p vortex --all-targets --all-features — passed.
• cargo clippy -p vortex --bench string_to_arrow --all-features — passed.
• cargo +nightly fmt --all -- --check — passed.
• git diff --check — passed.
• string_to_arrow --test — all 4 scenarios passed.
• to_arrow nested_string_export --test — all 24 scenarios passed.

[ClSlaid]

/cc @robert3005 This PR should restores the performance of #8902, please take a look.

And this patch should not be an end, it is just a quick fix, not yet perfed to leave no stone unturned.

Current matcher abstraction surely still has room to improve, but not in this PR.

[joseph-isaacs]

You should be able to execute away the filter and slice before you export? Why did you want to avoid this?

If you really need implementing a execute parent (filter, varbin) -> varbin kernel would make sense?

Or do you really to fuse filter + arrow export?

[ClSlaid]

You should be able to execute away the filter and slice before you export? Why did you want to avoid this?

None of them are avoided. execute_until will execute until ArrayRef is not wrapped inside those operators.

The matcher was too tight and it will make Fsst, Zstd or other types run through the canonical execute_until path first.

Sadly those pathes decode and write into VarBinView only, and then we have to copy them into VarBin, row by row. That's way too slow.

Meanwhile we already have direct decode implementations, it's fast and has less memory footprint, all in ArrayRef's append_to_builder. In short it can decode into reserved VarBinBuilder's buffer directly.

[robert3005]

I think the fix here might be to not do this execution and just rely on builders

[ClSlaid]

I've ignored some scenarios like Take(Filter(FSST)), I'll convert to draft and reopen once I'm ready.

[robert3005]

I think the general solution looks like #9371. I have ported the benchmarks to that stack fwiw. Thanks for pointing this out