[SPARK-27049][SQL] Support handling partition values in the abstraction of file source V2 #23961
Conversation
|
Test build #103002 has finished for PR 23961 at commit
|
|
retest this please. |
|
Test build #103021 has finished for PR 23961 at commit
|
| @@ -29,13 +29,3 @@ class PartitionRecordReader[T]( | |||
|
|
|||
| override def close(): Unit = rowReader.close() | |||
| } | |||
|
|
|||
| class PartitionRecordReaderWithProject[X, T]( | |||
There was a problem hiding this comment.
can we make this class to append partition values?
I feel it's weird to have these 3 methods in FilePartitionReaderFactory: buildReader, buildReaderWithPartitionValues and buildColumnarReaderWithPartitionValues. They are not symmetrical.
There was a problem hiding this comment.
Which class do you mean? I think it is bit complex to refactor the current ColumnarBatch to support appending partition values. I should create another JIRA for it. What do you think?
There was a problem hiding this comment.
it's just different ways to abstract the code: 1) add common methods in the parent class 2) create util classes.
In this case I feel 2) is better, because the added methods in the parent class are not symmetrical.
|
Close this one and create #23987 |
What changes were proposed in this pull request?
In
FileFormat, the methodbuildReaderWithPartitionValuesappends the partition values to the end of the result returned bybuildReader, so that for data sources like CSV/JSON/AVRO only need to implementbuildReaderto read a single file without taking care of partition values.This PR proposes to support handling partition values in file source v2 abstraction by:
buildReaderandbuildReaderWithPartitionValuesinFilePartitionReaderFactory, which have exactly the same meaning as they are inFileFormatbuildColumnarReaderasbuildColumnarReaderWithPartitionValuesto make the naming consistent.How was this patch tested?
Existing unit tests.