-
Notifications
You must be signed in to change notification settings - Fork 28.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[SPARK-8952] [SPARKR] - Wrap normalizePath calls with suppressWarnings #8343
Conversation
Jenkins, ok to test |
Test build #41337 has finished for PR 8343 at commit
|
The change looks fine to me, but I am not sure we reached a consensus on the JIRA. The main problem is that right now Also a minor procedural point --- Could you title the pull request |
For a short term solution, I think we can just use suppressWarnings(normalizePath(path)) to suppress the warning as is done in other places. For a long term solution, we need to discuss if we really need this helping feature to normalize a path, as there is no such feature in Scala API. If we want this, need to differentiate and handle local paths and HDFS paths. |
Test build #41709 has finished for PR 8343 at commit
|
Wrap normalizedPath calls with suppressWarnings to avoid issues with s3 and other specific paths.
Test build #41715 has finished for PR 8343 at commit
|
Thanks @lresende -- LGTM |
Jenkins, retest this please |
@yu-iskw I actually an error in Jenkins which says. I guess this is from the PR that added na.omit to the NAMESPACE yesterday
|
@shivaram I see. I'll investigate the cause. Thank you for letting me know. |
I think I found the problem. Our
I'll send a PR in a minute |
Test build #41718 has finished for PR 8343 at commit
|
Jenkins, retest this please |
Test build #41735 has finished for PR 8343 at commit
|
lint-r looks fine, Merging this. |
This is based on davies comment on SPARK-8952 which suggests to only call normalizePath() when path starts with '~' Author: Luciano Resende <lresende@apache.org> Closes #8343 from lresende/SPARK-8952. (cherry picked from commit 499e8e1) Signed-off-by: Shivaram Venkataraman <shivaram@cs.berkeley.edu>
This is based on @davies comment on SPARK-8952 which suggests to only call normalizePath() when path starts with '~'