[C++][Parquet] "Data size too large" error with byte-stream-split encoded data since Arrow 12.0.0

[adamreeve]

Describe the bug, including details regarding any error messages, version, and platform.

Arrow 12.0.0 has a regression where it can crash when reading byte-stream split encoded data written by itself or older versions of Arrow:

import numpy as np
import pyarrow as pa
import pyarrow.parquet as pq

x = pa.array(np.linspace(0.0, 1.0, 1_000_000), type=pa.float32())
table = pa.Table.from_arrays([x], names=['x'])
pq.write_table(table, 'data.parquet', use_dictionary=False, use_byte_stream_split=True)

table = pq.read_table('data.parquet')
print(table)

This crashes with:

Traceback (most recent call last):
  File "/home/.../write_read_data.py", line 9, in <module>
	table = pq.read_table('data.parquet')
			^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adam/.local/share/virtualenvs/arrow12-nieuEBn0/lib64/python3.11/site-packages/pyarrow/parquet/core.py", line 2986, in read_table
	return dataset.read(columns=columns, use_threads=use_threads,
		   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/adam/.local/share/virtualenvs/arrow12-nieuEBn0/lib64/python3.11/site-packages/pyarrow/parquet/core.py", line 2614, in read
	table = self._dataset.to_table(
			^^^^^^^^^^^^^^^^^^^^^^^
  File "pyarrow/_dataset.pyx", line 546, in pyarrow._dataset.Dataset.to_table
  File "pyarrow/_dataset.pyx", line 3449, in pyarrow._dataset.Scanner.to_table
  File "pyarrow/error.pxi", line 144, in pyarrow.lib.pyarrow_internal_check_status
  File "pyarrow/error.pxi", line 115, in pyarrow.lib.check_status
OSError: Data size too large for number of values (padding in byte stream split data page?)

But the above code works fine with pyarrow 11.0.0 and and 10.0.1.

It appears that #34140 caused this regression. I tested building pyarrow on the current main branch (commit 42d42b1) and could reproduce the error, but it was fixed after I reverted the merge of that PR (commit c31fb46).

Component(s)

Parquet

[adamreeve]

If I run this in a debugger, I can see ByteStreamSplitDecoder<DType>::SetData is called 4 times. For the first 3 times, num_values is 262,144 and len is 1,048,576, then on the 4th time len is again 1,048,576 but num_values is only 213,568 (for a total of 1,000,000 values).

If I only write 262,144 values, or a multiple of that, then everything works. But writing 262,145 values causes the crash.

From reading some of the context in #15173, it looks like the assumption that the encoders don't add any padding was incorrect?

[mapleFU]

I'll try to reproduce this problem. In ByteStreamSplitDecoder::SetData, it expects that num_values is greater or equals to the num of values in ByteStreamSplit, so it expects num_values * static_cast<int64_t>(sizeof(T)) >= len. It would be == if there is no null value in the page

I've reproduce this issue using PyArrow 12.0. I'll find out why.

[mapleFU]

Seems that using ParquetTableReader to just read that file is ok, and size as 100_000 or 200_000 is ok, but 300_000 would failed. I guess ParquetFileFormat with a huge page size is not ok.

[mapleFU]

I've found out the reason, I think it's not a bug in ByteStreamSplit, it's caused by page.size(), page reuse one buffer when ReadNextPage, and it size might be greater than expect size( page.uncompressed_size() ). However, ByteStreamSplit could expose that problem because it checks the boundary