[SPARK-41101][PYTHON][PROTOBUF] Message classname support for PYSPARK-PROTOBUF #38603
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
SandishKumarHN
force-pushed
the
classname-support-pyspark-protobuf
branch
4 times, most recently
from
30a3a66
to
640fb5b
Compare
SandishKumarHN
force-pushed
the
classname-support-pyspark-protobuf
branch
from
640fb5b
to
dc5f7b0
Compare
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
reviewed
Nov 11, 2022
HyukjinKwon
approved these changes
Nov 11, 2022
SandishKumarHN
force-pushed
the
classname-support-pyspark-protobuf
branch
from
ec8a9ca
to
e886ada
Compare
rangadi
reviewed
Nov 11, 2022
| the protobuf message name to look for in descriptor file. | ||
| messageName: str, optional | ||
| the protobuf message name to look for in descriptor file. Or | ||
| The Protobuf class name. E.g. ``org.spark.examples.protobuf.ExampleEvent``, |
There was a problem hiding this comment.
We need to add a comment about shading requirement. In scala, I am changing this name to shadedMessageClassName. See scaladoc update in open PR here: https://github.com/apache/spark/pull/38384/files#diff-93adb6d1143432aff776baed60d056133658635a2d8f9d29d925018fada64ff7R76-R83
rangadi
reviewed
Nov 11, 2022
| @@ -32,7 +32,7 @@ | |||
| def from_protobuf( | |||
| data: "ColumnOrName", | |||
| messageName: str, | |||
| descFilePath: str, | |||
| descFilePath: Optional[str] = None, | |||
There was a problem hiding this comment.
Update documentation to reflect change in contract.
@HyukjinKwon is it common to use Optional[] for optional arguments in PySpark?
|
Can one of the admins verify this patch? |
|
Merged to master. |
SandishKumarHN
added a commit
to SandishKumarHN/spark
that referenced
this pull request
Dec 12, 2022
…-PROTOBUF ### What changes were proposed in this pull request? - Adding message classname support for pyspark-protobuf - functions from_protobuf and to_protobuf ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How was this patch tested? - Python Doctests. - To avoid adding an extra jar, I used the existing spark-protobuf jar because it has a shadded google-protobuf and used a Timestamp message as an example protobuf messageClassName in Doctests rangadi HyukjinKwon Closes apache#38603 from SandishKumarHN/classname-support-pyspark-protobuf. Authored-by: SandishKumarHN <sanysandish@gmail.com> Signed-off-by: Hyukjin Kwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
Why are the changes needed?
Does this PR introduce any user-facing change?
How was this patch tested?
@rangadi @HyukjinKwon