You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I am reading an arrow::ListArray of Decimal128 from FixedLengthBinaryArray[16] in parquets and see on flamegraph that RawBytesToDecimalBytes consumes large amount of time and mostly due to page faults.
when arrow allocates new buffer for decimals and then parses FLBA into it.
As FLBA type has the same amount of memory per element i guess that this function should reuse given array data.
Possible easy(but very dirty) solution could be static_cast given array to Decimal array (as it does not have additional fields - it wouldn't be an error), const_cast raw_values of it and reverse byte order in-place
Another benefit is reducing memory usage of reading them
Component(s)
C++, Parquet
The text was updated successfully, but these errors were encountered:
Hattonuri
changed the title
Reuse memory of passed array in ConvertToDecimal
[C++][Parquet] Reuse memory of passed array in ConvertToDecimal
Nov 26, 2023
Describe the enhancement requested
I am reading an arrow::ListArray of Decimal128 from FixedLengthBinaryArray[16] in parquets and see on flamegraph that RawBytesToDecimalBytes consumes large amount of time and mostly due to page faults.
I assume that the problem happens here
arrow/cpp/src/parquet/arrow/reader_internal.cc
Lines 559 to 576 in eb5de18
when arrow allocates new buffer for decimals and then parses FLBA into it.
As FLBA type has the same amount of memory per element i guess that this function should reuse given array data.
Possible easy(but very dirty) solution could be static_cast given array to Decimal array (as it does not have additional fields - it wouldn't be an error), const_cast raw_values of it and reverse byte order in-place
Another benefit is reducing memory usage of reading them
Component(s)
C++, Parquet
The text was updated successfully, but these errors were encountered: