Search before asking
Paimon version
master, 2788fe596 (2.1-SNAPSHOT).
Compute Engine
Flink and Spark, wherever Paimon's own vectorized Parquet reader is used on a file it did not write: a format table over a Parquet directory, or a table produced by migrate_table / clone, which renames the source files in place rather than rewriting them.
Minimal reproduce step
Read a Parquet file whose DECIMAL column is stored on the BINARY physical type, with dictionary encoding off and at least two rows:
MessageType schema =
new MessageType(
"root",
Types.optional(PrimitiveTypeName.BINARY)
.as(LogicalTypeAnnotation.decimalType(2, 5))
.named("price"));
// three rows: unscaled 100, 200, 300, written with .withDictionaryEncoding(false)Reading it as DECIMAL(5, 2) throws on the second row:
java.lang.ArrayIndexOutOfBoundsException: 1
at org.apache.paimon.format.parquet.reader.VectorizedPlainValuesReader.readBinary(VectorizedPlainValuesReader.java:324)
at org.apache.paimon.format.parquet.reader.ParquetVectorUpdaterFactory$BinaryToDecimalUpdater.readValue(...)
at org.apache.paimon.format.parquet.reader.VectorizedParquetRecordReader.nextBatch(VectorizedParquetRecordReader.java:238)
BinaryToDecimalUpdater allocates a scratch vector of capacity 1 and then indexes it with the target row's offset:
this.bytesVector = new HeapBytesVector(1);
...
valuesReader.readBinary(1, bytesVector, offset);
BigInteger value = new BigInteger(bytesVector.getBytes(offset).getBytes());
HeapBytesVector.putByteArray writes start[elementNum], and start has one slot, so offset 1 is out of bounds. Only a dictionary-encoded page escapes it, because that path goes through decodeSingleDictionaryId and never touches the scratch vector.
What doesn't meet your expectations?
The Parquet spec allows DECIMAL on BINARY, and this reader has an updater for it, so a file that uses it should read. Instead any batch with two or more rows fails, and the message says nothing about decimals or about the file.
Anything else?
Paimon's own writer emits decimals as INT32, INT64 or FIXED_LEN_BYTE_ARRAY (ParquetSchemaConverter), never BINARY, so this only shows up on externally written files. parquet-avro maps an Avro bytes field carrying a decimal logical type onto BINARY, which is what Debezium and Kafka Connect produce.
The updater was introduced in #5582 (d83c5caf8), which is also when the scratch vector appeared. The tests added there generate random precisions, so they only ever exercise the INT32/INT64/FIXED_LEN_BYTE_ARRAY updaters and this class has had no coverage.
Are you willing to submit a PR?
Search before asking
Paimon version
master,
2788fe596(2.1-SNAPSHOT).Compute Engine
Flink and Spark, wherever Paimon's own vectorized Parquet reader is used on a file it did not write: a format table over a Parquet directory, or a table produced by
migrate_table/ clone, which renames the source files in place rather than rewriting them.Minimal reproduce step
Read a Parquet file whose DECIMAL column is stored on the BINARY physical type, with dictionary encoding off and at least two rows:
Reading it as
DECIMAL(5, 2)throws on the second row:BinaryToDecimalUpdaterallocates a scratch vector of capacity 1 and then indexes it with the target row's offset:HeapBytesVector.putByteArraywritesstart[elementNum], andstarthas one slot, so offset 1 is out of bounds. Only a dictionary-encoded page escapes it, because that path goes throughdecodeSingleDictionaryIdand never touches the scratch vector.What doesn't meet your expectations?
The Parquet spec allows DECIMAL on BINARY, and this reader has an updater for it, so a file that uses it should read. Instead any batch with two or more rows fails, and the message says nothing about decimals or about the file.
Anything else?
Paimon's own writer emits decimals as INT32, INT64 or FIXED_LEN_BYTE_ARRAY (
ParquetSchemaConverter), never BINARY, so this only shows up on externally written files. parquet-avro maps an Avrobytesfield carrying a decimal logical type onto BINARY, which is what Debezium and Kafka Connect produce.The updater was introduced in #5582 (
d83c5caf8), which is also when the scratch vector appeared. The tests added there generate random precisions, so they only ever exercise the INT32/INT64/FIXED_LEN_BYTE_ARRAY updaters and this class has had no coverage.Are you willing to submit a PR?