Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[C++] Output batch size control in ExecPlan #15155

Open
westonpace opened this issue Jan 2, 2023 · 0 comments
Open

[C++] Output batch size control in ExecPlan #15155

westonpace opened this issue Jan 2, 2023 · 0 comments

Comments

@westonpace
Copy link
Member

Describe the enhancement requested

Currently the exec plan processes things in fairly small (32Ki rows) batches according to the morsel/batch model. However, users may not want batches that small to be returned from Acero.

It shouldn't be too difficult to add an option to specify an output batch size and then, in the sink, to merge batches into larger output batches.

At the moment that would introduce a memory copy. In the longer term, we could investigate approaches that avoid this memory copy, but that could be addressed beyond this issue.

Component(s)

C++

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

1 participant