[SPARK-53805][SQL] Push Variant into DSv2 scan #52522
Conversation
|
cc @peter-toth , too. |
| hadoopFsRelation@HadoopFsRelation(_, _, _, _, _: ParquetFileFormat, _), _)) => | ||
| rewritePlan(p, projectList, filters, relation, hadoopFsRelation) | ||
| case p@PhysicalOperation(projectList, filters, relation: DataSourceV2Relation) => | ||
| rewriteV2RelationPlan(p, projectList, filters, relation.output, relation) |
There was a problem hiding this comment.
if we are sending the relation already do we need to send the relation.output seperately ?
There was a problem hiding this comment.
I overlooked this. Removed.
| SchemaPruning, | ||
| GroupBasedRowLevelOperationScanPlanning, | ||
| V1Writes, | ||
| PushVariantIntoScan, |
There was a problem hiding this comment.
now PushVariantIntoScan runs before the PruneFileSourcePartition, which i think was for v1 sources, does this matter or if i were to ask did we just like add in later, just because it was a new rule ?
There was a problem hiding this comment.
I don't think variant columns will ever be used in the partition schema. Schema transformations by PushVariantIntoScan shouldn't affect partition pruning in v1 sources.
| relation @ LogicalRelationWithTable( | ||
| hadoopFsRelation@HadoopFsRelation(_, _, _, _, _: ParquetFileFormat, _), _)) => | ||
| rewritePlan(p, projectList, filters, relation, hadoopFsRelation) | ||
| case p@PhysicalOperation(projectList, filters, relation: DataSourceV2Relation) => |
There was a problem hiding this comment.
Is there any code we can share between the v1 rewritePlan and the v2 rewriteV2RelationPlan?
There was a problem hiding this comment.
Yes, there’s shared logic. I intentionally left the v1 rewritePlan unchanged in this PR to keep the diff small and easier to review. After this merges, I’ll do a small follow-up to have v1 rewritePlan reuse the common code. If you prefer, I can fold that refactor into this PR.
There was a problem hiding this comment.
it's actually harder to review as I can't tell what's the key difference between the v1 and v2 versions with the current PR...
There was a problem hiding this comment.
Sorry for the confusion. I have updated the code.
The logic for transforming variant columns to struct is identical between DSv1 and DSv2. Now they both use the same helper methods (collectAndRewriteVariants, buildAttributeMap, buildFilterAndProject).
The only difference is how the transformed schema is communicated to the data source. DSv1 stores the new schema in HadoopFsRelation.dataSchema and the file source reads this field directly; DSv2 has no schema field to update. The schema is communicated later when V2ScanRelationPushDown calls pruneColumns.
What changes were proposed in this pull request?
Push Variant into DSv2 scan
Why are the changes needed?
with the change, DSV2 scan only needs to fetch the necessary shredded columns required by the plan
Does this PR introduce any user-facing change?
No
How was this patch tested?
new tests
Was this patch authored or co-authored using generative AI tooling?
No