New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-18910][SQL]Resolve faile to use UDF that jar file in hdfs. #16324
Conversation
@@ -2373,6 +2373,7 @@ class SparkContext(config: SparkConf) extends Logging { | |||
* various Spark features. | |||
*/ | |||
object SparkContext extends Logging { | |||
URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory()) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
SparkContext
might not be the good place. object SharedState
might be better.
In addition, you also need to add comments to explain why you did it before this line.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for you comment, I will add it to SharedState.
Please add the description in the JIRA to the PR description. FYI, you still can edit the description after you create the PR. |
Test build #70296 has finished for PR 16324 at commit
|
|
This might not be the right solution to do it, as explained above. |
Should we download the UDF jar from hdfs. |
First, I am not sure whether we should support reading UDF jar from HDFS. cc @rxin Second, if we want to support it, the best reviewers are @zsxwing @tdas They added the file HDFSMetadataLog.scala recently |
Currently,we can create a UDF with jar in HDFS, but failed to use it. |
This is to allow using jars defined using HDFS-API, not just HDFS right? In that case it sounds like a good idea too ... but we need a test case for it. |
I‘m sorry, @rxin, I don't understand what you mean. |
I was saying we need to create a test case for this change. |
(gentle ping @shenh062326 ) |
in spark 2.1.0 |
## What changes were proposed in this pull request? Spark 2.2 is going to be cut, it'll be great if SPARK-12868 can be resolved before that. There have been several PRs for this like [PR#16324](apache#16324) , but all of them are inactivity for a long time or have been closed. This PR added a SparkUrlStreamHandlerFactory, which relies on 'protocol' to choose the appropriate UrlStreamHandlerFactory like FsUrlStreamHandlerFactory to create URLStreamHandler. ## How was this patch tested? 1. Add a new unit test. 2. Check manually. Before: throw an exception with " failed unknown protocol: hdfs" <img width="914" alt="screen shot 2017-03-17 at 9 07 36 pm" src="https://cloud.githubusercontent.com/assets/8546874/24075277/5abe0a7c-0bd5-11e7-900e-ec3d3105da0b.png"> After: <img width="1148" alt="screen shot 2017-03-18 at 11 42 18 am" src="https://cloud.githubusercontent.com/assets/8546874/24075283/69382a60-0bd5-11e7-8d30-d9405c3aaaba.png"> Author: Weiqing Yang <yangweiqing001@gmail.com> Closes apache#17342 from weiqingy/SPARK-18910.
## What changes were proposed in this pull request? Spark 2.2 is going to be cut, it'll be great if SPARK-12868 can be resolved before that. There have been several PRs for this like [PR#16324](#16324) , but all of them are inactivity for a long time or have been closed. This PR added a SparkUrlStreamHandlerFactory, which relies on 'protocol' to choose the appropriate UrlStreamHandlerFactory like FsUrlStreamHandlerFactory to create URLStreamHandler. ## How was this patch tested? 1. Add a new unit test. 2. Check manually. Before: throw an exception with " failed unknown protocol: hdfs" <img width="914" alt="screen shot 2017-03-17 at 9 07 36 pm" src="https://cloud.githubusercontent.com/assets/8546874/24075277/5abe0a7c-0bd5-11e7-900e-ec3d3105da0b.png"> After: <img width="1148" alt="screen shot 2017-03-18 at 11 42 18 am" src="https://cloud.githubusercontent.com/assets/8546874/24075283/69382a60-0bd5-11e7-8d30-d9405c3aaaba.png"> Author: Weiqing Yang <yangweiqing001@gmail.com> Closes #17342 from weiqingy/SPARK-18910. (cherry picked from commit 2ba1eba) Signed-off-by: Marcelo Vanzin <vanzin@cloudera.com>
What changes were proposed in this pull request?
When I create a UDF that jar file in hdfs, I can't use the UDF.
The reason is when org.apache.spark.sql.internal.SessionState.FunctionResourceLoader.loadResource, the uri.toURL throw exception with " failed unknown protocol: hdfs"
I think we should setURLStreamHandlerFactory method on URL with an instance of FsUrlStreamHandlerFactory, just like:
How was this patch tested?
I have test it in my cluster.