[SPARK-32824][Core] Improve the error message when the user forgets the .amount in a resource config #29685
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
.amount after a resource like spark.executor.resource.gpu
tgravescs
changed the title
|
Test build #128421 has finished for PR 29685 at commit
|
HyukjinKwon
approved these changes
Sep 9, 2020
|
Merged to master and branch-3.0. |
HyukjinKwon
pushed a commit
that referenced
this pull request
Sep 9, 2020
…he .amount in a resource config ### What changes were proposed in this pull request? If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing: ``` ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1967) at ``` This makes it so we have a readable error thrown ### Why are the changes needed? confusing error for users ### Does this PR introduce _any_ user-facing change? just error message ### How was this patch tested? Tested manually on standalone cluster Closes #29685 from tgravescs/SPARK-32824. Authored-by: Thomas Graves <tgraves@nvidia.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit e8634d8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
|
@tgravescs, it's just nothing but my curiosity, why don't you add your NVIDIA email into your GitHub account so the commit links your GitHub account :-)? |
|
no particular reason, thanks for pointing it out. |
|
Yeah, there was no conflict and tests passed (e86d90b). |
|
great thanks |
holdenk
pushed a commit
to holdenk/spark
that referenced
this pull request
Oct 27, 2020
…he .amount in a resource config ### What changes were proposed in this pull request? If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing: ``` ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range: -1 at java.lang.String.substring(String.java:1967) at ``` This makes it so we have a readable error thrown ### Why are the changes needed? confusing error for users ### Does this PR introduce _any_ user-facing change? just error message ### How was this patch tested? Tested manually on standalone cluster Closes apache#29685 from tgravescs/SPARK-32824. Authored-by: Thomas Graves <tgraves@nvidia.com> Signed-off-by: HyukjinKwon <gurwls223@apache.org> (cherry picked from commit e8634d8) Signed-off-by: HyukjinKwon <gurwls223@apache.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What changes were proposed in this pull request?
If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing:
This makes it so we have a readable error thrown
Why are the changes needed?
confusing error for users
Does this PR introduce any user-facing change?
just error message
How was this patch tested?
Tested manually on standalone cluster