Conversation
| @@ -145,7 +134,17 @@ impl VTable for ConstantVTable { | |||
| let scalar_value = ScalarValue::from_proto_bytes(bytes, dtype)?; | |||
| let scalar = Scalar::try_new(dtype.clone(), scalar_value)?; | |||
There was a problem hiding this comment.
This could be on the GPU and there still very slow to copy
There was a problem hiding this comment.
I believe we talked about this before, I think we don't care right now? cc @gatesn @onursatici
There was a problem hiding this comment.
I don't think we care right now.
We need to figure out the right flat layout logic for GPU + host behavior. Small constant arrays should store their values in the metadata. Larger constant arrays and list arrays should store their values in an array.
This would be using ScalarValue::Array.
The thing we need to figure out is a way for ConstantArray to store this array value in a child array, rather than serialized into its own metadata. If we do this, then list-constants and large constant scalars will remain on the GPU all the way through deserialization.
There was a problem hiding this comment.
Note that ScalarValue::Array does not exist yet (we want to have it), and this is blocking that
connortsui20
force-pushed
the
ct/constant-scalar
branch
from
0a6cc3b to
f336e6a
Compare
Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
connortsui20
force-pushed
the
ct/constant-scalar
branch
from
f336e6a to
5867160
Compare
## Summary Similar in nature to #6751. We want to be able to work with in-memory metadata more easily, and we also want to make sure all scalar deserialization happens is `deserialize` instead of `build`. ## API Changes Renames the constructors to be in line with the rest of the encodings. ## Testing Existing tests should be sufficient. --------- Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
## Summary Similar in nature to #6751 and #6759. We want to be able to work with in-memory metadata more easily, and we also want to make sure all scalar deserialization happens is `deserialize` instead of `build`. This is the last of the changes to metadata needed to unblock array-valued scalars in metadata. ## Testing Existing tests should be sufficient. Signed-off-by: Connor Tsui <connor.tsui20@gmail.com>
3 participants
Summary
Changes the
MetadataforConstantArrayto be aScalar. Note that this is not a breaking change since theMetadatais purely an in-memory construct.Testing
All of the existing tests should be sufficient.