-
Notifications
You must be signed in to change notification settings - Fork 28k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-44719][SQL][3.5] Fix NoClassDefFoundError when using Hive UDF #42447
Conversation
<groupId>org.codehaus.jackson</groupId> | ||
<artifactId>jackson-core-asl</artifactId> | ||
<version>${codehaus.jackson.version}</version> | ||
<scope>${hadoop.deps.scope}</scope> |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The master branch uses Hadoop 3.3.6. So Hadoop doesn't need this.
The branch-3.5 uses Hadoop 3.3.4. so Hadoop still need this.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Do you think we can have a test coverage, @wangyum ?
f7090f0
to
befa483
Compare
The test cannot cover it because jackson-mapper-asl is in the classpath: https://github.com/apache/spark/blob/branch-3.5/pom.xml#L1825-L1829 |
### What changes were proposed in this pull request? This PR partially reverts SPARK-43225. Added jackson-core-asl and jackson-mapper-asl back to pre-built distributions. ### Why are the changes needed? Fix `NoClassDefFoundError` when using Hive UDF: ``` spark-sql (default)> add jar /Users/yumwang/Downloads/HiveUDFs-1.0-SNAPSHOT.jar; Time taken: 0.413 seconds spark-sql (default)> CREATE TEMPORARY FUNCTION long_to_ip as 'net.petrabarus.hiveudfs.LongToIP'; Time taken: 0.038 seconds spark-sql (default)> SELECT long_to_ip(2130706433L) FROM range(10); 23/08/08 20:17:58 ERROR SparkSQLDriver: Failed in [SELECT long_to_ip(2130706433L) FROM range(10)] java.lang.NoClassDefFoundError: org/codehaus/jackson/map/type/TypeFactory at org.apache.hadoop.hive.ql.udf.UDFJson.<clinit>(UDFJson.java:64) at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:348) ... ``` ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? manual test. Closes #42447 from wangyum/SPARK-44719-branch-3.5. Authored-by: Yuming Wang <yumwang@ebay.com> Signed-off-by: Kent Yao <yao@apache.org>
Thanks, merged to branch-3.5. |
the GitHub micro 'close' seems not working, close it manually |
What changes were proposed in this pull request?
This PR partially reverts SPARK-43225. Added jackson-core-asl and jackson-mapper-asl back to pre-built distributions.
Why are the changes needed?
Fix
NoClassDefFoundError
when using Hive UDF:Does this PR introduce any user-facing change?
No.
How was this patch tested?
manual test.