[SPARK-27698][SQL] Add new method convertibleFilters for getting pushed down filters in Parquet file reader
#24597
Conversation
|
mark this as WIP before #24598 is merged. |
gengliangwang
changed the title
|
Test build #105376 has finished for PR 24597 at commit
|
gengliangwang
force-pushed
the
refactorParquetFilters
branch
from
75fe737
to
fa8f48c
Compare
gengliangwang
force-pushed
the
refactorParquetFilters
branch
from
fa8f48c
to
b22ea80
Compare
gengliangwang
changed the title
|
Test build #105527 has finished for PR 24597 at commit
|
|
Test build #105529 has finished for PR 24597 at commit
|
|
This is ready for review @dongjoon-hyun @wangyum @rdblue @cloud-fan |
dongjoon-hyun
changed the title
|
Thank you for pinging me, @gengliangwang . Shall we wait for one day? Currently, after SPARK-27699, |
|
retest this please. |
|
@dongjoon-hyun @cloud-fan Please review this, so that we can continue the migration of Parquet V2. |
|
Test build #105613 has finished for PR 24597 at commit
|
gengliangwang
changed the title
convertibleFilters for getting pushed down filters in Parquet file reader
|
thanks, merging to master! |
| /** | ||
| * Returns a map, which contains parquet field name and data type, if predicate push down applies. | ||
| */ | ||
| private def getFieldMap(dataType: MessageType): Map[String, ParquetField] = { |
There was a problem hiding this comment.
@gengliangwang, don't have to move codes around to make it easier to track ...
|
Looks fine but can you clarify the relation between convertibleFilters and createFilters? |
What changes were proposed in this pull request?
To return accurate pushed filters in Parquet file scan(#24327 (review)), we can process the original data source filters in the following way:
For "And" operators, split the conjunctive predicates and try converting each of them. After that
1.1 if partially predicate pushed down is allowed, return convertible results;
1.2 otherwise, return the whole predicate if convertible, or empty result if not convertible.
For "Or" operators, if both children can be pushed down, it is partially or totally convertible; otherwise, return empty result
For other operators, they are not able to be partially pushed down.
2.1 if the entire predicate is convertible, return itself
2.2 otherwise, return an empty result.
This PR also contains code refactoring. Currently
ParquetFilters. createFilteraccepts parameterschema: MessageTypeand create field mapping for every input filter. We can make it a class member and avoid creating thenameToParquetFieldmapping for every input filter.How was this patch tested?
Unit test