You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Arrow arrays can share the same backing store, even if the array is just a "view" of a slice of another array.
Yet, when flight_data_from_arrow_batch encodes the arrays into a FlightData, it blindly copies the entire buffer ready to be sent over the wire.
Thus, for example, when DataFusion uses the arrow::compute::limit operator to return a few elements of an array, we still end up with a the full (potentially) large array being sent over the wire.
Since encoding the array in a FlightData involves copying the data anyway, perhaps it would be beneficial to take the Array length in consideration and copy only the parts of the buffer that contain actual data.
Arrow arrays can share the same backing store, even if the array is just a "view" of a slice of another array.
Yet, when
flight_data_from_arrow_batch
encodes the arrays into a FlightData, it blindly copies the entire buffer ready to be sent over the wire.Thus, for example, when DataFusion uses the
arrow::compute::limit
operator to return a few elements of an array, we still end up with a the full (potentially) large array being sent over the wire.Since encoding the array in a FlightData involves copying the data anyway, perhaps it would be beneficial to take the Array length in consideration and copy only the parts of the buffer that contain actual data.
Reporter: Marko Mikulicic
Note: This issue was originally created as ARROW-12265. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: