[SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively#25778
[SPARK-29080][CORE][SPARKR] Support R file extension case-insensitively#25778Loquats wants to merge 4 commits intoapache:masterfrom
Conversation
dongjoon-hyun
changed the title
dongjoon-hyun
left a comment
dongjoon-hyun
left a comment
There was a problem hiding this comment.
To check the coding style quickly, you can use dev/scalastyle in your PR directory. Please try to run it and fix the errors. Thanks.
|
@dongjoon-hyun Thanks for the tip. I ran The tests for these projects pass. Please let me know if there is anything else I should test. |
|
Thank you for update. Let's trigger a full test. |
|
ok to test |
dongjoon-hyun
changed the title
|
@Loquats . BTW, do you have Apache JIRA account? You can create an Apache Spark JIRA issue there. Please file a JIRA issue and use the JIRA ID in the PR title next time. For this issue, I created one for you and updated this PR title. |
|
@dongjoon-hyun Thanks for creating the JIRA issue. I just created an Apache JIRA account, and I will remember to create a JIRA issue next time. My username is andyzhang. |
|
Test build #110575 has finished for PR 25778 at commit
|
|
Thank you for informing your JIRA id. When this PR is merged, your ID will be added to Apache Spark contributor group and SPARK-29080 will be assigned to you. |
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala
Outdated
Show resolved
Hide resolved
resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/Client.scala
Outdated
Show resolved
Hide resolved
dongjoon-hyun
changed the title
|
Test build #110601 has finished for PR 25778 at commit
|
|
I updated the PR description for the others because this will become a commit log. |
|
You are added to the Apache Spark contributor group and SPARK-29080 is assigned to you. |
|
Thanks all for reviewing this PR! |
### What changes were proposed in this pull request? Make r file extension check case insensitive for spark-submit. ### Why are the changes needed? spark-submit does not accept `.r` files as R scripts. Some codebases have r files that end with lowercase file extensions. It is inconvenient to use spark-submit with lowercase extension R files. The error is not very clear (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L232). ``` $ ./bin/spark-submit examples/src/main/r/dataframe.r Exception in thread "main" org.apache.spark.SparkException: Cannot load main class from JAR file:/Users/dongjoon/APACHE/spark-release/spark-2.4.4-bin-hadoop2.7/examples/src/main/r/dataframe.r ``` ### Does this PR introduce any user-facing change? Yes. spark-submit can now be used to run R scripts with `.r` file extension. ### How was this patch tested? Manual. ``` $ mv examples/src/main/r/dataframe.R examples/src/main/r/dataframe.r $ ./bin/spark-submit examples/src/main/r/dataframe.r ``` Closes apache#25778 from Loquats/r-case. Authored-by: Andy Zhang <yue.zhang@databricks.com> Signed-off-by: Dongjoon Hyun <dhyun@apple.com> Change-Id: I57ca6068c04ac3cdfd8606dbb48d5192bff1749d Reviewed-on: https://bigdataoss-internal-review.googlesource.com/c/third_party/apache/spark/+/17710 Reviewed-by: Igor Dvorzhak <idv@google.com>
5 participants
What changes were proposed in this pull request?
Make r file extension check case insensitive for spark-submit.
Why are the changes needed?
spark-submit does not accept
.rfiles as R scripts. Some codebases have r files that end with lowercase file extensions. It is inconvenient to use spark-submit with lowercase extension R files. The error is not very clear (https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/deploy/SparkSubmitArguments.scala#L232).Does this PR introduce any user-facing change?
Yes. spark-submit can now be used to run R scripts with
.rfile extension.How was this patch tested?
Manual.