You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If RecordBatch.length is less than array length, the reader should ignore the portion of the array(s) beyond RecordBatch.length. This will allow partially populated batches to be read in scenarios identified in the above discussion.
Status GetFieldMetadata(int field_index, ArrayData* out) {
auto nodes = metadata_->nodes();
// pop off a fieldif (field_index >= static_cast<int>(nodes->size())) {
returnStatus::Invalid("Ran out of field metadata, likely malformed");
}
const flatbuf::FieldNode* node = nodes->Get(field_index);
* //out->length = node->length();*
* out->length = metadata_->length();*
out->null_count = node->null_count();
out->offset = 0;
returnStatus::OK();
}
Attached is a test IPC File containing a batch with length 1, array length 3.
Wes McKinney / @wesm:
I think that the fields' length needs to be respected in GetFieldMetadata (since this can vary for nested fields). The appropriate fix is probably here:
0.13 ignored RecordBatch.length. 0.14 requires that RecordBatch.length and array length be equal. As per https://lists.apache.org/thread.html/2692dd8fe09c92aa313bded2f4c2d4240b9ef75a8604ec214eb02571@%3Cdev.arrow.apache.org%3E , we discussed changing this so that RecordBatch.length can be [0,array length].
If RecordBatch.length is less than array length, the reader should ignore the portion of the array(s) beyond RecordBatch.length. This will allow partially populated batches to be read in scenarios identified in the above discussion.
Attached is a test IPC File containing a batch with length 1, array length 3.
Reporter: John Muehlhausen / @jgm-ktg
Original Issue Attachments:
PRs and other links:
Note: This issue was originally created as ARROW-5916. Please see the migration documentation for further details.
The text was updated successfully, but these errors were encountered: