Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Arrow buffer Size Option #7784

Merged
merged 27 commits into from
Jun 11, 2023
Merged

Arrow buffer Size Option #7784

merged 27 commits into from
Jun 11, 2023

Conversation

pdet
Copy link
Contributor

@pdet pdet commented Jun 1, 2023

This PR allows users to set the Arrow Buffer Size used for Strings, BLOBS, UUIDS.

It can be set by changing the arrow_large_buffer_size option.

SET arrow_large_buffer_size=True

By default, it also changes DuckDB to output regular buffer sizes.

Obs: Currently, the extension building CI is failing. The Arrow-IPC extension PR duckdb/arrow#12 should be merged first, and the extension retagged to fix all the extension CI.

OBS2: I consider this somewhat of a bug fix, but I can retag it to feature if desired.

Copy link
Contributor

@Tishj Tishj left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me, could maybe use a couple extra tests since this changes a lot of different places where arrow is produced.

Of the top of my head, a scalar UDF test that makes sure the strings passed to the UDF are of the correct type could be nice

@carlopi
Copy link
Contributor

carlopi commented Jun 6, 2023

Hi @pdet!
I guess WebAssembly CI will fail, given that the is an incompatibility (chicken & egg, ...), I will do the (minor) changes WebAssembly side, I'd say this looks stable, and then we merge the two PRs them more or less together (so subsequent CI will not fail).

@Mytherin Mytherin merged commit 44aeb23 into duckdb:master Jun 11, 2023
86 checks passed
@Mytherin
Copy link
Collaborator

Thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants