Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[SPARK-47703][SQL] Modify the simpleString of DataSourceV2ScanRelation to distinguish it from DataSourceV2Relation #45829

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

Zouxxyy
Copy link
Contributor

@Zouxxyy Zouxxyy commented Apr 3, 2024

What changes were proposed in this pull request?

Modify the simpleString of DataSourceV2ScanRelation to distinguish it from DataSourceV2Relation

Why are the changes needed?

At present, the simpleString of DataSourveV2ScanRelation and DataSourveV2Relation are the same, it is difficult to distinguish them.

before:

=== Applying Rule org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown ===
 Subquery true                                                                                                                                 Subquery true
!+- RelationV2[cs_sold_date_sk#116L, cs_sold_time_sk#117L, cs_ship_date_sk#118L, cs_bill_customer_sk#119L, ... 30 more fields] catalog_sales   +- RelationV2[cs_sold_date_sk#116L, cs_sold_time_sk#117L, cs_ship_date_sk#118L, cs_bill_customer_sk#119L, ... 30 more fields] catalog_sales
 

after:

=== Applying Rule org.apache.spark.sql.execution.datasources.v2.V2ScanRelationPushDown ===
 Subquery true                                                                                                                                 Subquery true
!+- RelationV2[cs_sold_date_sk#116L, cs_sold_time_sk#117L, cs_ship_date_sk#118L, cs_bill_customer_sk#119L, ... 30 more fields] catalog_sales   +- ScanRelationV2[cs_sold_date_sk#116L, cs_sold_time_sk#117L, cs_ship_date_sk#118L, cs_bill_customer_sk#119L, ... 30 more fields] catalog_sales
 

Does this PR introduce any user-facing change?

How was this patch tested?

Was this patch authored or co-authored using generative AI tooling?

… to distinguish it from DataSourceV2Relation
@github-actions github-actions bot added the SQL label Apr 3, 2024
@HyukjinKwon
Copy link
Member

what's diff between before and after?

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Apr 3, 2024

V2ScanRelationPushDown

before:
after apply V2ScanRelationPushDown, RelationV2 -> RelationV2, we don't know if this rule works

after:
RelationV2 -> ScanRelationV2

@HyukjinKwon
Copy link
Member

That's not how it's written in the PR description :-).

@Zouxxyy
Copy link
Contributor Author

Zouxxyy commented Apr 3, 2024

That's not how it's written in the PR description :-).

Oh, this is the standard log of spark.sql.planChangeLog.rules, you need to slide it
image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
2 participants