-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Arrow buffer Size Option #7784
Arrow buffer Size Option #7784
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Looks good to me, could maybe use a couple extra tests since this changes a lot of different places where arrow is produced.
Of the top of my head, a scalar UDF test that makes sure the strings passed to the UDF are of the correct type could be nice
Hi @pdet! |
…thod of parallel reader
Thanks! |
This PR allows users to set the Arrow Buffer Size used for Strings, BLOBS, UUIDS.
It can be set by changing the
arrow_large_buffer_size
option.By default, it also changes DuckDB to output regular buffer sizes.
Obs: Currently, the extension building CI is failing. The Arrow-IPC extension PR duckdb/arrow#12 should be merged first, and the extension retagged to fix all the extension CI.
OBS2: I consider this somewhat of a bug fix, but I can retag it to feature if desired.