You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently the exec plan processes things in fairly small (32Ki rows) batches according to the morsel/batch model. However, users may not want batches that small to be returned from Acero.
It shouldn't be too difficult to add an option to specify an output batch size and then, in the sink, to merge batches into larger output batches.
At the moment that would introduce a memory copy. In the longer term, we could investigate approaches that avoid this memory copy, but that could be addressed beyond this issue.
Component(s)
C++
The text was updated successfully, but these errors were encountered:
Describe the enhancement requested
Currently the exec plan processes things in fairly small (32Ki rows) batches according to the morsel/batch model. However, users may not want batches that small to be returned from Acero.
It shouldn't be too difficult to add an option to specify an output batch size and then, in the sink, to merge batches into larger output batches.
At the moment that would introduce a memory copy. In the longer term, we could investigate approaches that avoid this memory copy, but that could be addressed beyond this issue.
Component(s)
C++
The text was updated successfully, but these errors were encountered: