Support empty projection in ParquetRecordBatchReader
#1560
Conversation
| @@ -214,12 +205,6 @@ impl ParquetRecordBatchReader { | |||
| batch_size: usize, | |||
| array_reader: Box<dyn ArrayReader>, | |||
| ) -> Result<Self> { | |||
| // Check that array reader is struct array reader | |||
| array_reader | |||
There was a problem hiding this comment.
This check does not seem necessary, all it cares about is that it yields StructArray
| /// but with row counts that correspond to the amount of data in the file | ||
| /// | ||
| /// This is useful for when projection eliminates all columns within a collection | ||
| pub fn make_empty_array_reader(row_count: usize) -> Box<dyn ArrayReader> { |
There was a problem hiding this comment.
I don't like this name, but it was the best I could come up with
| } | ||
|
|
||
| struct EmptyArrayReader { | ||
| data_type: ArrowType, |
There was a problem hiding this comment.
data_type seems not necessary to keep here, it is always ArrowType::Struct(vec![]).
There was a problem hiding this comment.
It's needed for get_data_type(&self) -> &ArrowType as it needs to return a reference with a lifetime coupled to the object itself.
There was a problem hiding this comment.
yea, just found it and want to go to comment. you already replied. nvm.
Codecov Report
@@ Coverage Diff @@
## master #1560 +/- ##
==========================================
- Coverage 82.86% 82.85% -0.01%
==========================================
Files 190 191 +1
Lines 55057 55085 +28
==========================================
+ Hits 45622 45642 +20
- Misses 9435 9443 +8
Continue to review full report at Codecov.
|
alamb
changed the title
ParquetRecordBatchReader
Which issue does this PR close?
Closes #1537
Rationale for this change
See ticket
What changes are included in this PR?
See ticket
Are there any user-facing changes?
No, the only changes are to experimental modules.