[SPARK-48794][CONNECT] df.mergeInto support for Spark Connect (Scala and Python) #46960
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
HyukjinKwon
pushed a commit
that referenced
this pull request
Jul 3, 2024
### What changes were proposed in this pull request? Spark 4.0 added a new `df.mergeInto` API, but it is missing from PySpark. This PR fixes that. The support for this API in Spark Connect Python API will be added later by #46960. ### Why are the changes needed? Because PySpark does not support `df.mergeInto`. ### Does this PR introduce _any_ user-facing change? Yes, the user would be able to use the `df.mergeInto` API. ### How was this patch tested? New unit tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes #47086 from xupefei/pyspark-mergeinto. Authored-by: Paddy Xu <xupaddy@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
xupefei
changed the title
|
@HyukjinKwon and @grundprinzip Could you review this PR? Thanks! |
xupefei
changed the title
HyukjinKwon
changed the title
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
HyukjinKwon
reviewed
Jul 3, 2024
|
|
||
| // (Required) Whether to enable schema evolution. | ||
| bool with_schema_evolution = 7; | ||
| } |
There was a problem hiding this comment.
This would need some reviews from @hvanhovell and/or @grundprinzip
|
cc @zhengruifeng and @ueshin too |
HyukjinKwon
approved these changes
Jul 10, 2024
ericm-db
pushed a commit
to ericm-db/spark
that referenced
this pull request
Jul 10, 2024
### What changes were proposed in this pull request? Spark 4.0 added a new `df.mergeInto` API, but it is missing from PySpark. This PR fixes that. The support for this API in Spark Connect Python API will be added later by apache#46960. ### Why are the changes needed? Because PySpark does not support `df.mergeInto`. ### Does this PR introduce _any_ user-facing change? Yes, the user would be able to use the `df.mergeInto` API. ### How was this patch tested? New unit tests. ### Was this patch authored or co-authored using generative AI tooling? No. Closes apache#47086 from xupefei/pyspark-mergeinto. Authored-by: Paddy Xu <xupaddy@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
|
@xupefei mind resolving conflicts? I will just merge |
Done! |
zhengruifeng
approved these changes
Jul 12, 2024
|
Merged to master. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
This PR introduces
df.mergeIntosupport for Spark Connect Scala and Python clients.This work contains four components:
MergeIntoTableCommandand expressionMergeAction.MergeIntoCommands.MetgeIntoWriterthat allows users to build merges.MetgeIntoWriterthat allows users to build merges.Components 3 and 4 and independent to each other. They both depends on Component 1.
Why are the changes needed?
We need to increase the functionality of Spark Connect to be on par with Classic.
Does this PR introduce any user-facing change?
Yes, new Dataframe APIs are introduced.
How was this patch tested?
Added new tests.
Was this patch authored or co-authored using generative AI tooling?
No.