-
Notifications
You must be signed in to change notification settings - Fork 792
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add Field::size
and DataType::size
#3147
Labels
arrow
Changes to the arrow crate
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Comments
crepererum
added
the
enhancement
Any new improvement worthy of a entry in the changelog
label
Nov 21, 2022
crepererum
added a commit
to crepererum/arrow-rs
that referenced
this issue
Nov 21, 2022
Add a way to calculate in-memory size of `Field` and `DataType`. Closes apache#3147.
crepererum
added a commit
to crepererum/arrow-rs
that referenced
this issue
Nov 21, 2022
Add a way to calculate in-memory size of `Field` and `DataType`. Closes apache#3147.
crepererum
added a commit
to crepererum/arrow-rs
that referenced
this issue
Nov 21, 2022
Add a way to calculate in-memory size of `Field` and `DataType`. Closes apache#3147.
tustvold
pushed a commit
that referenced
this issue
Nov 22, 2022
|
alamb
added
parquet
Changes to the parquet crate
arrow
Changes to the arrow crate
labels
Nov 25, 2022
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
arrow
Changes to the arrow crate
enhancement
Any new improvement worthy of a entry in the changelog
parquet
Changes to the parquet crate
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
In DataFusion, it would be nice to know how much data is allocated so we can bail out early instead of OOMing (a slight over-allocation is OK, so this can be measured after the fact). For that purpose, it would be nice to know how much memory a specific instance of
Field
/DataType
requires.Describe the solution you'd like
Add:
Note that fields and data types have a cyclic dependency, so we probably need to implement both in a single PR.
Describe alternatives you've considered
Not adding this feature to arrow but let downstream users (e.g. DataFusion) do this.
Additional context
-
The text was updated successfully, but these errors were encountered: