Skip to content

Commit

Permalink
[SPARK-32824][CORE] Improve the error message when the user forgets t…
Browse files Browse the repository at this point in the history
…he .amount in a resource config

### What changes were proposed in this pull request?

If the user forgets to specify .amount on a resource config like spark.executor.resource.gpu, the error message thrown is very confusing:

```
ERROR SparkContext: Error initializing SparkContext.java.lang.StringIndexOutOfBoundsException: String index out of range:
-1 at java.lang.String.substring(String.java:1967) at
```

This makes it so we have a readable error thrown

### Why are the changes needed?

confusing error for users
### Does this PR introduce _any_ user-facing change?

just error message

### How was this patch tested?

Tested manually on standalone cluster

Closes #29685 from tgravescs/SPARK-32824.

Authored-by: Thomas Graves <tgraves@nvidia.com>
Signed-off-by: HyukjinKwon <gurwls223@apache.org>
  • Loading branch information
tgravescs authored and HyukjinKwon committed Sep 9, 2020
1 parent 96ff87d commit e8634d8
Showing 1 changed file with 6 additions and 1 deletion.
Original file line number Diff line number Diff line change
Expand Up @@ -149,7 +149,12 @@ private[spark] object ResourceUtils extends Logging {

def listResourceIds(sparkConf: SparkConf, componentName: String): Seq[ResourceID] = {
sparkConf.getAllWithPrefix(s"$componentName.$RESOURCE_PREFIX.").map { case (key, _) =>
key.substring(0, key.indexOf('.'))
val index = key.indexOf('.')
if (index < 0) {
throw new SparkException(s"You must specify an amount config for resource: $key " +
s"config: $componentName.$RESOURCE_PREFIX.$key")
}
key.substring(0, index)
}.distinct.map(name => new ResourceID(componentName, name))
}

Expand Down

0 comments on commit e8634d8

Please sign in to comment.