New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-6081] Support fetching http/https uris in driver runner. #4832
Conversation
Test build #28123 has finished for PR 4832 at commit
|
Test failure doesn't look anything to do with my change? |
Unrelated it seems. This would require a JIRA. I'm not sure about how important this is vs passing the SecurityManager around one more layer, but in any event the change is not big. I defer to others for an opinion. |
I'm curious too about SecurityManager, let me know what makes sense about it. |
retest this please |
@@ -468,6 +468,7 @@ private[spark] object Utils extends Logging { | |||
in: InputStream, | |||
destFile: File, | |||
fileOverwrite: Boolean): Unit = { | |||
logDebug(s"Creating temp file in ${destFile.getParentFile.getAbsolutePath}") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we should add this
Looks good. |
Test build #29126 has finished for PR 4832 at commit
|
I'm merging this into master after fixing the comments myself. Thanks. |
Currently if passed uris such as http/https, it won't able to fetch them as it only calls HadoopFs get.
This fix utilizes the existing util method to fetch remote uris as well.