fix(reader): resolve sparse Utf8/Binary rows lazily, and stop dropping the fill

CHANGELOG.md

@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - A `vortex.sequence` column no longer materializes `base + i * multiplier` into a full buffer on decode; rows are computed on access, so the encoding allocates nothing regardless of row count — closing an `OutOfMemoryError` risk from a metadata-only encoding whose row count no buffer bounds. ([#335](https://github.com/dfa1/vortex-java/issues/335))
 - A primitive `vortex.dict` column decoded through the encoding path no longer expands its codes into an `n * elemSize` buffer; it now returns the same lazy `DictXxxArray` carriers the layout path already used, so a dict column keeps the dictionary's memory benefit however it is reached. ([#336](https://github.com/dfa1/vortex-java/issues/336))
 - A `vortex.patched` column with no patches no longer allocates and copies a full duplicate of its inner child; the child is aliased directly when it already covers every row. ([#337](https://github.com/dfa1/vortex-java/issues/337))
+- A sparse Utf8/Binary column (`vortex.sparse`) with a non-null string or binary fill no longer renders every unpatched row as the empty string: the fill value was dropped on the way into the decode, so only the patched rows were ever right. ([#340](https://github.com/dfa1/vortex-java/issues/340))
+- A sparse Utf8/Binary column no longer merges its patches into a fresh `length`-row bytes buffer plus an `(n + 1)` offsets table on decode; rows resolve to the fill or a patch on access, so the column costs its patches instead of its rows. ([#340](https://github.com/dfa1/vortex-java/issues/340))
 
 ## [0.13.1] — 2026-08-06
 

docs/compatibility.md

@@ -92,7 +92,7 @@ integer width is wire-legal, mirroring `VarBinArray`), and `ScanIterator` could
 | `vortex.alp`                | `AlpEncodingDecoder`             | `AlpEncodingEncoder`             | ✅      | ✅      | F64, F32                                                              |
 | `vortex.alprd`              | `AlpRdEncodingDecoder`           | `AlpRdEncodingEncoder`           | ✅      | ✅      | F64, F32                                                              |
 | `vortex.dict`               | `DictEncodingDecoder`            | `DictEncodingEncoder`            | ✅      | ✅      | Primitive, Utf8/Binary                                                |
-| `vortex.sparse`             | `SparseEncodingDecoder`          | `SparseEncodingEncoder`          | ✅      | ✅      | Primitive                                                             |
+| `vortex.sparse`             | `SparseEncodingDecoder`          | `SparseEncodingEncoder`          | ✅      | ✅      | Primitive, Bool, Utf8/Binary (decode); Primitive (encode)             |
 | `vortex.sequence`           | `SequenceEncodingDecoder`        | `SequenceEncodingEncoder`        | ✅      | ✅      | Primitive                                                             |
 | `vortex.struct`             | `StructEncodingDecoder`          | `StructEncodingEncoder`          | ✅      | ✅      | Struct                                                                |
 | `vortex.chunked`            | `ChunkedEncodingDecoder`         | `ChunkedEncodingEncoder`         | ✅      | ✅      | Primitive + Struct concat                                             |
@@ -144,7 +144,7 @@ decoder falls into one of three shapes:
 | `vortex.alp`                | Lazy          | Lazy          | `LazyAlpXxxArray`; broadcast → `LazyConstantXxxArray`; patched stays Materialized, ADR 0010 + 0015 |
 | `vortex.alprd`              | Lazy          | Lazy          | `LazyAlpRdDoubleArray`/`LazyAlpRdFloatArray` — left/right + patches on access |
 | `vortex.dict`               | Lazy          | Lazy          | `DictXxxArray` (numeric) + `VarBinDictArray` (string), ADR 0012          |
-| `vortex.sparse`             | Lazy          | Lazy          | `LazySparseXxxArray` (primitive + bool); Utf8/Binary stays Materialized, ADR 0015 |
+| `vortex.sparse`             | Lazy          | Lazy          | `LazySparseXxxArray` (primitive + bool) + `VarBinSparseArray` (Utf8/Binary); fill broadcast, patch resolved per access; patch-free range → `VarBinConstantArray`, ADR 0015 |
 | `vortex.sequence`           | Lazy          | Lazy          | `LazySequenceXxxArray`; `base + i * multiplier` per access, no buffer, ADR 0015 |
 | `vortex.struct`             | Zero-copy     | Zero-copy     | `StructArray` wraps fields                                               |
 | `vortex.chunked`            | Lazy          | Lazy          | `ChunkedXxxArray` (primitive/Bool) + `VarBinChunkedArray` (Utf8/Binary), ADR 0012 |

integration/src/test/java/io/github/dfa1/vortex/integration/NullSparseRunEndInteropIntegrationTest.java

@@ -14,13 +14,16 @@
 import io.github.dfa1.vortex.reader.array.MaskedArray;
 import io.github.dfa1.vortex.reader.array.NullArray;
 import io.github.dfa1.vortex.reader.array.ShortArray;
+import io.github.dfa1.vortex.reader.array.VarBinArray;
+import io.github.dfa1.vortex.reader.array.VarBinSparseArray;
 import org.apache.arrow.c.ArrowArray;
 import org.apache.arrow.c.ArrowSchema;
 import org.apache.arrow.c.Data;
 import org.apache.arrow.memory.BufferAllocator;
 import org.apache.arrow.vector.BigIntVector;
 import org.apache.arrow.vector.Float8Vector;
 import org.apache.arrow.vector.SmallIntVector;
+import org.apache.arrow.vector.VarCharVector;
 import org.apache.arrow.vector.VectorSchemaRoot;
 import org.apache.arrow.vector.types.FloatingPointPrecision;
 import org.apache.arrow.vector.types.pojo.ArrowType;
@@ -30,6 +33,7 @@
 import org.junit.jupiter.api.io.TempDir;
 
 import java.io.IOException;
+import java.nio.charset.StandardCharsets;
 import java.nio.file.Path;
 import java.util.ArrayList;
 import java.util.HashMap;
@@ -48,10 +52,11 @@
 ///
 /// The bundled `vortex-jni` compressor is version-pinned, so its encoding choice for a crafted
 /// input is deterministic — verified by probing: a mostly-null column with a small fraction of
-/// scattered non-`ALP`-able values compresses to `vortex.sparse` with a null fill scalar, and a
-/// column of distinct value runs interleaved with null runs compresses to `vortex.runend` with
-/// null run-values. Both assertions below therefore hard-check the chosen encoding; a future JNI
-/// bump that changes the choice is a deliberate change that should refresh this fixture.
+/// scattered non-`ALP`-able values compresses to `vortex.sparse` with a null fill scalar (for
+/// utf8 too, not only for primitives), and a column of distinct value runs interleaved with null
+/// runs compresses to `vortex.runend` with null run-values. Every assertion below therefore
+/// hard-checks the chosen encoding; a future JNI bump that changes the choice is a deliberate
+/// change that should refresh this fixture.
 class NullSparseRunEndInteropIntegrationTest {
 
     private static final Session SESSION = Session.create();
@@ -119,6 +124,41 @@ void jniNullFillSparse_i64_nullRowsDecodeNull(@TempDir Path tmp) throws IOExcept
         assertThat(result).containsExactly(expected);
     }
 
+    /// The utf8 sibling of the two null-fill sparse cases above, and the ground-truth cover for
+    /// the lazy `VarBinSparseArray` carrier (#340) — before it, this shape merged every patch
+    /// into a full `length`-row bytes buffer, and the fill was dropped on the way in.
+    ///
+    /// Only the null-fill half is reachable from here: probing the pinned JNI compressor with a
+    /// utf8 column that is mostly one NON-null value gets `vortex.dict` (or `vortex.fsst` over
+    /// `vortex.runend` for a mostly-empty-string column), never sparse with a non-null string
+    /// fill. So the non-null-fill fix stays unit-covered only until a corpus file or a JNI bump
+    /// produces that shape.
+    @Test
+    void jniNullFillSparse_utf8_nullRowsDecodeNull(@TempDir Path tmp) throws IOException {
+        // Given — a nullable utf8 column, ~99% null with a few scattered distinct strings.
+        Schema schema = new Schema(List.of(Field.nullable("v", new ArrowType.Utf8())));
+        String[] expected = new String[ROWS];
+        Path file = tmp.resolve("null_fill_sparse_utf8.vtx");
+        writeJni(file, schema, (root, i) -> {
+            VarCharVector vec = (VarCharVector) root.getVector("v");
+            if (i % 97 == 0) {
+                String value = "x" + i;
+                expected[i] = value;
+                vec.setSafe(i, value.getBytes(StandardCharsets.UTF_8));
+            } else {
+                vec.setNull(i);
+            }
+        });
+
+        // When
+        List<String> result = readUtf8(file);
+
+        // Then — the column decodes through the lazy sparse carrier, and every row round-trips.
+        assertThat(usedEncodings(file)).contains("vortex.sparse");
+        assertThat(carrier(file)).isEqualTo(VarBinSparseArray.class);
+        assertThat(result).containsExactly(expected);
+    }
+
     @Test
     void jniNullRunRunEnd_i16_nullRowsDecodeNull(@TempDir Path tmp) throws IOException {
         // Given — the uci-online-retail `customerid` u16? shape: runs of a distinct value per
@@ -254,6 +294,27 @@ private static List<Long> readI64(Path file) throws IOException {
         return out;
     }
 
+    private static List<String> readUtf8(Path file) throws IOException {
+        var out = new ArrayList<String>();
+        forEachValue(file, (masked, inner, i) -> out.add(masked.isValid(i) ? ((VarBinArray) inner).getString(i) : null));
+        return out;
+    }
+
+    /// Returns the concrete [Array] class the single column's payload decodes to, so a test can
+    /// assert the decode path taken and not only the values it produced — the eager and lazy
+    /// paths agree on every value here, so values alone would pass on both.
+    ///
+    /// @param file the Vortex file to scan
+    /// @return the runtime class of the first chunk's unwrapped payload array
+    /// @throws IOException if the file cannot be opened or scanned
+    private static Class<?> carrier(Path file) throws IOException {
+        try (VortexReader reader = VortexReader.open(file, ReadRegistry.loadAll());
+             var iter = reader.scan(ScanOptions.all())) {
+            MaskedArray masked = iter.next().column("v");
+            return masked.inner().getClass();
+        }
+    }
+
     private static List<Short> readI16(Path file) throws IOException {
         var out = new ArrayList<Short>();
         forEachValue(file, (masked, inner, i) -> out.add(masked.isValid(i) ? ((ShortArray) inner).getShort(i) : null));

reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinArray.java

@@ -14,8 +14,10 @@
 /// Implementations: [VarBinOffsetArray] for standard offset-based layout, [VarBinDictArray] for
 /// dictionary-encoded strings, [VarBinChunkedArray] for multi-chunk columns, [VarBinViewArray]
 /// for Arrow StringView / BinaryView layout (16-byte view per row + zero or more shared data
-/// buffers), [VarBinSlicedArray] for a zero-copy row-shifted view, and [VarBinConstantArray] for
-/// a `vortex.constant` value broadcast across every row. All accessors resolve transparently
+/// buffers), [VarBinSlicedArray] for a zero-copy row-shifted view, [VarBinConstantArray] for
+/// a `vortex.constant` value broadcast across every row, [VarBinRunEndArray] for a
+/// `vortex.runend` column resolved run-by-run, and [VarBinSparseArray] for a `vortex.sparse`
+/// column resolved patch-by-patch. All accessors resolve transparently
 /// regardless of implementation; only [VarBinOffsetArray] exposes
 /// [VarBinOffsetArray#offsetsSegment()] and [VarBinOffsetArray#offsetsPtype()].
 ///

reader/src/main/java/io/github/dfa1/vortex/reader/array/VarBinSparseArray.java

@@ -0,0 +1,115 @@
+package io.github.dfa1.vortex.reader.array;
+
+import io.github.dfa1.vortex.core.model.DType;
+
+import java.lang.foreign.MemorySegment;
+import java.nio.charset.StandardCharsets;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.function.IntConsumer;
+
+/// Lazy Sparse-encoded [VarBinArray]: `getBytes(i) = patchValues[binSearch(i + offset)]`, or
+/// `fill` at every unpatched position.
+///
+/// The VarBin member of the `LazySparseXxxArray` family, built on the same
+/// `findPatch`/`walkPatches` helpers the primitive variants use. Where those broadcast a
+/// scalar fill, this one broadcasts the fill's raw bytes, so a sparse column costs
+/// `numPatches` values of storage rather than a merged `length`-row bytes buffer plus an
+/// `(length + 1)` offsets table.
+///
+/// The fill bytes matter beyond allocation: the eager merge this replaces described unpatched
+/// rows as zero-length ranges, which rendered every one of them as the empty string no matter
+/// what the fill scalar said. The Rust reference resolves an unpatched row to the fill value
+/// for utf8/binary exactly as it does for primitives.
+///
+/// [#forEachByteLength(IntConsumer)] walks patches in order (one binary search up front, then
+/// a per-patch step) so sequential reads are O(numPatches) work plus `length` emissions, not
+/// O(length x log(numPatches)).
+///
+/// [#bytesSegment()] is the [MemorySegment#NULL] sentinel and [#segmentIfPresent()] is empty:
+/// no single contiguous buffer holds the resolved rows, the same convention
+/// [VarBinChunkedArray], [VarBinRunEndArray], and [VarBinConstantArray] use. Consumers that
+/// need the flat bytes-plus-offsets shape get it on demand from
+/// [VarBinArray#toOffsetMode(VarBinArray, java.lang.foreign.SegmentAllocator)].
+///
+/// The `patchIndices` array is typed as [Array] because the indices ptype varies — backed by
+/// one of [ByteArray], [ShortArray], [IntArray], [LongArray].
+///
+/// A patch-free array is not represented here but as a [VarBinConstantArray] over the same
+/// fill bytes, which resolves in O(1) with no search at all; `patchValues` is therefore always
+/// non-null and non-empty.
+///
+/// @param dtype        logical element type (Utf8 or Binary)
+/// @param length       total logical row count
+/// @param fill         raw bytes of the fill scalar, shared by every unpatched row;
+///                     [#getBytes(long)] clones it per that method's copy contract
+/// @param patchValues  values for patched positions; length = `numPatches`
+/// @param patchIndices sorted absolute positions of patches; length = `numPatches`
+/// @param offset       starting absolute position; logical row `i` maps to absolute `i + offset`
+@SuppressWarnings("java:S6218") // internal data carrier; record components are arrays of immutable primitives or refs that flow through pipelines without ever being compared.
+public record VarBinSparseArray(DType dtype, long length, byte[] fill,
+                                VarBinArray patchValues, Array patchIndices, long offset)
+        implements VarBinArray {
+
+    /// No single contiguous segment backs the resolved rows.
+    ///
+    /// @return the [MemorySegment#NULL] sentinel
+    @Override
+    public MemorySegment bytesSegment() {
+        return MemorySegment.NULL;
+    }
+
+    /// No single contiguous segment backs the resolved rows.
+    ///
+    /// @return always empty
+    @Override
+    public Optional<MemorySegment> segmentIfPresent() {
+        return Optional.empty();
+    }
+
+    @Override
+    public byte[] getBytes(long i) {
+        int p = patch(i);
+        return p >= 0 ? patchValues.getBytes(p) : fill.clone();
+    }
+
+    @Override
+    public String getString(long i) {
+        int p = patch(i);
+        return p >= 0 ? patchValues.getString(p) : new String(fill, StandardCharsets.UTF_8);
+    }
+
+    @Override
+    public int getByteLength(long i) {
+        int p = patch(i);
+        return p >= 0 ? patchValues.getByteLength(p) : fill.length;
+    }
+
+    @Override
+    public void forEachByteLength(IntConsumer c) {
+        int fillLen = fill.length;
+        SparseArrays.walkPatches(patchIndices, patchValues.length(), offset, offset + length,
+                () -> c.accept(fillLen),
+                p -> c.accept(patchValues.getByteLength(p)));
+    }
+
+    /// Zero-copy truncation: only the row count shrinks, since rows are resolved through
+    /// `patchIndices` on read and trailing patches past the new end simply go unvisited.
+    ///
+    /// @param rows number of leading rows to keep
+    /// @return a length-`rows` view over the same patches
+    @Override
+    public VarBinArray limited(long rows) {
+        return rows >= length ? this
+                : new VarBinSparseArray(dtype, rows, fill, patchValues, patchIndices, offset);
+    }
+
+    /// Locates the patch at logical row `i`.
+    ///
+    /// @param i zero-based logical row index (must be in `[0, length)`)
+    /// @return the patch index, or `-1` when the row is unpatched
+    private int patch(long i) {
+        Objects.checkIndex(i, length);
+        return SparseArrays.findPatch(patchIndices, patchValues.length(), i + offset);
+    }
+}