In-Memory Table: Store Integer 32-bit and Integer 16-bit more efficiently. Add support for Float 32-bit.
#6109
Labels
-libs
Libraries: New libraries to be implemented
l-table-column-datatypes
p-lowest
Should be completed at some point
x-on-hold
Comments
8 tasks
radeusgd
added
p-lowest
radeusgd
changed the title
Integer 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floatsInteger 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floats.
radeusgd
changed the title
Integer 32-bit and Integer 16-bit more efficiently. Add support for 32-bit floats.Integer 32-bit and Integer 16-bit more efficiently. Add support for Float 32-bit.
mergify bot
pushed a commit
that referenced
this issue
Aug 22, 2023
… in-memory backend and check for ArithmeticOverflow in LongStorage (#7557) - Closes #5159 - Now data downloaded from the database can keep the type much closer to the original type (like string length limits or smaller integer types). - Cast also exposes these types. - The integers are still all stored as 64-bit Java `long`s, we just check their bounds. Changing underlying storage for memory efficiency may come in the future: #6109 - Fixes #7565 - Fixes #7529 by checking for arithmetic overflow in in-memory integer arithmetic operations that could overflow. Adds a documentation note saying that the behaviour for Database backends is unspecified and depends on particular database.
|
Subject to review when we look at Apache Arrow |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
With #5159 we can set our storage to accept integers of limited size.
For now however, we've been still storing them as 64-bit ints, because having alternative storages requires lots of boilerplate and there was not enough time to implement it.
This task is to actually attempt it and alter the storage to store 32-bit integers as Java
intarrays, and 16-bit asshortarrays.Similarly, it should add support for storing
Float 32-bitasfloatarrays (currently onlydoubleare supported).To do so and keep it maintainable, we may need to create a code generator, to avoid duplicating tons of code.
Value
The text was updated successfully, but these errors were encountered: