fix(reader): make constant Utf8/Binary decode lazy - #331
Merged
Conversation
ConstantEncodingDecoder.decodeString was the one vortex.constant value type without a LazyConstantXxxArray: every other type (primitive, bool, decimal) broadcasts a single stored value per row in O(1), but strings eagerly wrote n copies of the scalar into a real OffsetMode buffer (n * strLen allocation + copy), the only crash-adjacent risk that survived the RunEnd/Constant/Zoned/Pco hardening batch (a large n * strLen product could overflow negative or just OOM). Root cause: VarBinArray's only flat representation, OffsetMode, has no broadcast/modulo path the way AbstractMaterializedArray gives the primitive Materialized*Array types. Added VarBinArray.ConstantMode: a sealed-permitted record holding the scalar's bytes once, with every accessor returning it for any row — bytesSegment()/segmentIfPresent() follow the same "no single contiguous buffer" convention already used by ChunkedMode/ViewMode, so generic consumers still flatten it correctly via VarBinArray.toOffsetMode(). decodeString now builds this directly instead of an eager OffsetMode. docs/compatibility.md: vortex.constant's Notes column now credits VarBinArray.ConstantMode alongside LazyConstantXxxArray.
2 tasks
dfa1
added a commit
that referenced
this pull request
Aug 6, 2026
VarBinArray was the one Array sub-hierarchy still declared sealed with an explicit permits list, unlike its siblings (ByteArray, LongArray, DecimalArray, ...), which reopen the hierarchy via non-sealed and let each representation live as an ordinary top-level class. Nothing in the codebase does an exhaustive switch over VarBinArray's specific modes (grepped: zero matches), so the sealing bought no compiler-enforced exhaustiveness — only coupling the interface's declaration to the full list of its implementations, and forcing every new representation (most recently VarBinConstantArray, #331) to be nested inside VarBinArray.java and added to its permits clause. VarBinArray is now `non-sealed`. Each former nested record is a top-level class in reader.array, renamed to match the sibling families' self-describing naming (MaterializedByteArray, LazyConstantLongArray, ...): - OffsetMode -> VarBinOffsetArray - DictMode -> VarBinDictArray - ChunkedMode -> VarBinChunkedArray - ViewMode -> VarBinViewArray - SlicedMode -> VarBinSlicedArray - ConstantMode -> VarBinConstantArray checkedLength (bounds-check shared by the offset- and dict-backed representations) moved to a new package-private VarBinArrays helper, mirroring the RunEndArrays precedent, since it can no longer be a private interface method reachable from sibling top-level classes. Pure rename/move: no behavior change. All call sites across reader/writer/cli updated; docs/compatibility.md's Notes column updated to the new class names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ConstantEncodingDecoder.decodeStringwas the onevortex.constantvalue type without a lazy broadcast array: every other type (primitive, bool, decimal) returns a single stored value per row in O(1) via aLazyConstantXxxArray; strings eagerly wrotencopies of the scalar into a realOffsetModebuffer (n * strLenallocation + copy) — the one crash-adjacent gap left after the RunEnd/Constant/Zoned/Pco adversarial-input hardening batch (PR fix(reader): harden RunEnd/Constant/Zoned/Pco against malformed input #330).VarBinArray.ConstantMode: holds the scalar's bytes once, every accessor returns it for any row, no buffer allocated. Follows the same "no single contiguous segment" conventionChunkedMode/ViewModealready use, so generic consumers (e.g.VarBinArray.toOffsetMode) still flatten it correctly when they need a real buffer.decodeStringnow buildsConstantModedirectly instead of the eagerOffsetMode.docs/compatibility.mdupdated to creditVarBinArray.ConstantModealongsideLazyConstantXxxArrayforvortex.constant.Test plan
./mvnw -pl fsst,core,reader -am verify -DskipITs./mvnw -pl writer,integration -am verify(real-world files with constant Utf8/Binary columns)./mvnw -pl integration -am verify -Dit.test=DocsConsistencyTest -Dvortex.it.excludedGroups=VarBinArrayTest.Constant(accessor/copy/limited/bounds/toOffsetMode contracts) andConstantEncodingDecoderTest(decodes toConstantMode; a row count too large to ever eagerly materialize still decodes instantly)🤖 Generated with Claude Code