Search before asking
Paimon version
master, 475be566f (2.1-SNAPSHOT).
Compute Engine
Any engine reading Parquet through the vectorized reader, for a column encoded with DELTA_BYTE_ARRAY (Parquet v2) where the reader skips values, which happens when a row filter gives the reader a row-index list.
Minimal reproduce step
Skip an odd number of values in a DELTA_BYTE_ARRAY column, then skip again, then read. The value that comes back has NUL bytes where its prefix should be:
String[] vals = new String[] {"aaaa", "aaab", "aaac", "aaad"};
Utils.writeData(writer, vals);
reader.initFromPage(vals.length, writer.getBytes().toInputStream());
reader.skipBinary(1);
reader.skipBinary(1);
reader.readBinary(0); // "\0\0\0c" instead of "aaac"skipBinary alternates two vectors and leaves previous pointing at the buffer of whichever one it wrote last:
c1.reset();
...
c1.putByteArray(0, bytes, 0, length);
BytesColumnVector.Bytes b = c1.getBytes(0);
previous = ByteBuffer.wrap(b.data, b.offset, b.len);
After an odd number of skipped values previous points into tempBinaryValVector, and the next skipBinary call starts by resetting that same vector. HeapBytesVector.reset() zeroes the data buffer:
// We don't reset buffer to avoid unnecessary copy.
Arrays.fill(buffer, (byte) 0);
so the prefix copied out of previous right after is all zeros. No error is raised; the value is simply wrong.
What doesn't meet your expectations?
Two things. Reading a value back should not depend on how many values were skipped before it. And reset() should not wipe the data buffer at all: the comment one line above says it does not, every read is bounded by the start and length arrays that reset() does clear, and wiping costs O(buffer size) per batch on the vectorized read path. The Arrays.fill arrived with the reader rework in #4982, contradicting the comment it sits under.
Anything else?
What I did not establish is whether Paimon's own writer produces DELTA_BYTE_ARRAY (that needs the v2 writer) and whether readNextFilteredRowGroup is reached in a normal read. So the corruption may need an externally written v2 file plus a row filter; the encoding is supported on the read side either way.
Are you willing to submit a PR?
Search before asking
Paimon version
master,
475be566f(2.1-SNAPSHOT).Compute Engine
Any engine reading Parquet through the vectorized reader, for a column encoded with
DELTA_BYTE_ARRAY(Parquet v2) where the reader skips values, which happens when a row filter gives the reader a row-index list.Minimal reproduce step
Skip an odd number of values in a
DELTA_BYTE_ARRAYcolumn, then skip again, then read. The value that comes back has NUL bytes where its prefix should be:skipBinaryalternates two vectors and leavespreviouspointing at the buffer of whichever one it wrote last:After an odd number of skipped values
previouspoints intotempBinaryValVector, and the nextskipBinarycall starts by resetting that same vector.HeapBytesVector.reset()zeroes the data buffer:so the prefix copied out of
previousright after is all zeros. No error is raised; the value is simply wrong.What doesn't meet your expectations?
Two things. Reading a value back should not depend on how many values were skipped before it. And
reset()should not wipe the data buffer at all: the comment one line above says it does not, every read is bounded by thestartandlengtharrays thatreset()does clear, and wiping costs O(buffer size) per batch on the vectorized read path. TheArrays.fillarrived with the reader rework in #4982, contradicting the comment it sits under.Anything else?
What I did not establish is whether Paimon's own writer produces
DELTA_BYTE_ARRAY(that needs the v2 writer) and whetherreadNextFilteredRowGroupis reached in a normal read. So the corruption may need an externally written v2 file plus a row filter; the encoding is supported on the read side either way.Are you willing to submit a PR?