[SPARK-51831][SQL] Column pruning with existsJoin for Datasource V2 #51046
+106
−9
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
jackylee-ch
force-pushed
the
SPARK-51831
branch
6 times, most recently
from
280a301 to
3cbac75
Compare
LuciferYang
reviewed
Jun 3, 2025
|
|
||
| private def createScanBuilder(plan: LogicalPlan) = plan.transform { | ||
| case r @ DataSourceV2ScanRelation(relation, _, _, _, _) | ||
| if relation.getTagValue(V2_SCAN_BUILDER_HOLDER).nonEmpty => |
There was a problem hiding this comment.
When will the content of tags be removed?
LuciferYang
reviewed
Jun 3, 2025
| @@ -658,16 +658,55 @@ abstract class SchemaPruningSuite | |||
| |where not exists (select null from employees e where e.name.first = c.name.first | |||
| | and e.employer.name = c.employer.company.name) | |||
| |""".stripMargin) | |||
| // TODO: enable this check once we fix the schema pruning for V1 nested columns | |||
| /** | |||
There was a problem hiding this comment.
If V1 hasn't figured out how to fix it yet, perhaps we could temporarily check for different outcomes based on whether dataSourceName is included in the result of SQLConf.USE_V1_SOURCE_LIST?
This is still better than commenting out the assertion
LuciferYang
reviewed
Jun 3, 2025
| @@ -658,16 +658,55 @@ abstract class SchemaPruningSuite | |||
| |where not exists (select null from employees e where e.name.first = c.name.first | |||
| | and e.employer.name = c.employer.company.name) | |||
| |""".stripMargin) | |||
| // TODO: enable this check once we fix the schema pruning for V1 nested columns | |||
There was a problem hiding this comment.
TODO should be used with a JIRA ticket.
jackylee-ch
force-pushed
the
SPARK-51831
branch
2 times, most recently
from
a2c0714 to
0121f77
Compare
|
friendly ping @cloud-fan |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are the changes needed?
Recently, I have been testing TPC-DS queries based on DataSource V2, and noticed that column pruning does not occur in scenarios involving EXISTS (SELECT * FROM ... WHERE ...). As a result, the scan ends up reading all columns instead of just the required ones. This issue is reproducible in queries like Q10, Q16, Q35, Q69, and Q94.
This PR introduces
PostV2ScanRelationPushdownto address the column pruning issues that may arise after optimizer rules are applied.Below is the plan changes for the newly added test case.
Before this PR
After this PR
Does this PR introduce any user-facing change?
No.
How was this patch tested?
Newly added UT.
Was this patch authored or co-authored using generative AI tooling?
No.