You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
currently, shuffle writer keeps record batches uncompressed until they are spilled or flushed to next stage. if we compress them before spilling, we can reduce memory usage and the amount of spilled data.
Describe the solution you'd like
when a record batch is finished, instead of buffering it into a Vec<RecordBatch>, we compress it into aVec[u8] which is later spilled or flushed.
Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered:
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
currently, shuffle writer keeps record batches uncompressed until they are spilled or flushed to next stage. if we compress them before spilling, we can reduce memory usage and the amount of spilled data.
Describe the solution you'd like
when a record batch is finished, instead of buffering it into a
Vec<RecordBatch>
, we compress it into aVec[u8]
which is later spilled or flushed.Describe alternatives you've considered
Additional context
The text was updated successfully, but these errors were encountered: