Skip to content
Permalink
Browse files

[SPARK-24021][CORE] fix bug in BlacklistTracker's updateBlacklistForF…

…etchFailure

## What changes were proposed in this pull request?

There‘s a miswrite in BlacklistTracker's updateBlacklistForFetchFailure:
```
val blacklistedExecsOnNode =
    nodeToBlacklistedExecs.getOrElseUpdate(exec, HashSet[String]())
blacklistedExecsOnNode += exec
```
where first **exec** should be **host**.
## How was this patch tested?

adjust existed test.

Author: wuyi <ngone_5451@163.com>

Closes #21104 from Ngone51/SPARK-24021.

(cherry picked from commit 0deaa52)
Signed-off-by: Imran Rashid <irashid@cloudera.com>
  • Loading branch information...
Ngone51 authored and squito committed Apr 19, 2018
1 parent 32bec6c commit 7fb11176f285b4de47e61511c09acbbb79e5c44c
@@ -210,7 +210,7 @@ private[scheduler] class BlacklistTracker (
updateNextExpiryTime()
killBlacklistedExecutor(exec)

val blacklistedExecsOnNode = nodeToBlacklistedExecs.getOrElseUpdate(exec, HashSet[String]())
val blacklistedExecsOnNode = nodeToBlacklistedExecs.getOrElseUpdate(host, HashSet[String]())
blacklistedExecsOnNode += exec
}
}
@@ -574,6 +574,9 @@ class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfterEach with M
verify(allocationClientMock, never).killExecutors(any(), any(), any(), any())
verify(allocationClientMock, never).killExecutorsOnHost(any())

assert(blacklist.nodeToBlacklistedExecs.contains("hostA"))
assert(blacklist.nodeToBlacklistedExecs("hostA").contains("1"))

// Enable auto-kill. Blacklist an executor and make sure killExecutors is called.
conf.set(config.BLACKLIST_KILL_ENABLED, true)
blacklist = new BlacklistTracker(listenerBusMock, conf, Some(allocationClientMock), clock)
@@ -589,6 +592,8 @@ class BlacklistTrackerSuite extends SparkFunSuite with BeforeAndAfterEach with M
1000 + blacklist.BLACKLIST_TIMEOUT_MILLIS)
assert(blacklist.nextExpiryTime === 1000 + blacklist.BLACKLIST_TIMEOUT_MILLIS)
assert(blacklist.nodeIdToBlacklistExpiryTime.isEmpty)
assert(blacklist.nodeToBlacklistedExecs.contains("hostA"))
assert(blacklist.nodeToBlacklistedExecs("hostA").contains("1"))

// Enable external shuffle service to see if all the executors on this node will be killed.
conf.set(config.SHUFFLE_SERVICE_ENABLED, true)

0 comments on commit 7fb1117

Please sign in to comment.
You can’t perform that action at this time.