Describe the enhancement requested
The BYTE_STREAM_SPLIT encoder and decoder allocate a SmallVector to hold the addresses of the different streams. While this is valid for small number of streams (which is most or all legitimate use cases), very large numbers of streams (e.g. FLBA(100000000)) can trigger a huge temporary memory allocation.
This issue was found by OSS-Fuzz: https://issues.oss-fuzz.com/issues/511575321
Component(s)
C++, Parquet
Describe the enhancement requested
The BYTE_STREAM_SPLIT encoder and decoder allocate a
SmallVectorto hold the addresses of the different streams. While this is valid for small number of streams (which is most or all legitimate use cases), very large numbers of streams (e.g. FLBA(100000000)) can trigger a huge temporary memory allocation.This issue was found by OSS-Fuzz: https://issues.oss-fuzz.com/issues/511575321
Component(s)
C++, Parquet