Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

1.5.0 spark shaded protobuf version conflict with application #1188

Closed
hannin opened this issue Jan 7, 2024 · 2 comments
Closed

1.5.0 spark shaded protobuf version conflict with application #1188

hannin opened this issue Jan 7, 2024 · 2 comments

Comments

@hannin
Copy link

hannin commented Jan 7, 2024

Expected behavior

My application uses protobuf also with version higher than 3.19.x
with org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0, protobuf generated files makes error like

        if (!com.google.protobuf.GeneratedMessageV3.isStringEmpty(osVersion_)) {
                                                   ^
  symbol:   method isStringEmpty(java.lang.Object)
  location: class com.google.protobuf.GeneratedMessageV3

It's because protoc version and protobuf-java lib version not matched

Actual behavior

with org.apache.sedona:sedona-spark-shaded-3.0_2.12:1.5.0, custom protobuf-java version builds

Steps to reproduce the problem

I made sample project for reproduce https://oss.navercorp.com/tak/spark-sedona-test
./gradlew build failed there

Settings

Sedona version = 1.5.0

Apache Spark version = not explicit

Apache Flink version = no

API type = Scala

Scala version = 2.12

JRE version = 1.8

Python version = no

Environment = build phase


with shaded jar, there is relocate function to fix this (ref https://maven.apache.org/plugins/maven-shade-plugin/examples/class-relocation.html)

@jiayuasu
Copy link
Member

jiayuasu commented Jan 7, 2024

@hannin Since you are using gradle, you can use our unshaded jar: https://sedona.apache.org/1.5.0/setup/maven-coordinates/#use-sedona-unshaded-jars

Here is an example Scala project that uses the unshaded jar. You can exclude protobuf-java from the unshaded dependency. https://github.com/apache/sedona/blob/master/examples/spark-sql/pom.xml

@hannin
Copy link
Author

hannin commented Jan 7, 2024

oh there was documented solution already.. thank you

@hannin hannin closed this as completed Jan 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants