Skip to content
Permalink
Browse files

[SPARK-26082][MESOS][FOLLOWUP] Add UT on fetcher cache option on Meso…

…sClusterScheduler

## What changes were proposed in this pull request?

This patch adds UT on testing SPARK-26082 to avoid regression. While #23743 reduces the possibility to make a similar mistake, the needed lines of code for adding tests are not that huge, so I guess it might be worth to add them.

## How was this patch tested?

Newly added UTs. Test "supports setting fetcher cache" fails when #23743 is not applied and succeeds when #23743 is applied.

Closes #23744 from HeartSaVioR/SPARK-26082-add-unit-test.

Authored-by: Jungtaek Lim (HeartSaVioR) <kabhwan@gmail.com>
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
(cherry picked from commit b4e1d14)
Signed-off-by: Dongjoon Hyun <dhyun@apple.com>
  • Loading branch information...
HeartSaVioR authored and dongjoon-hyun committed Feb 7, 2019
1 parent bb6dbd2 commit 3abf45d6f3568d801460f558cb66f31260413a12
@@ -254,6 +254,54 @@ class MesosClusterSchedulerSuite extends SparkFunSuite with LocalSparkContext wi
assert(networkInfos.get(0).getLabels.getLabels(1).getValue == "val2")
}

test("supports setting fetcher cache") {
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "true"))

val mem = 1000
val cpu = 1

val response = scheduler.submitDriver(
new MesosDriverDescription("d1", "jar", mem, cpu, true,
command,
Map("spark.mesos.executor.home" -> "test",
"spark.app.name" -> "test"),
"s1",
new Date()))

assert(response.success)

val offer = Utils.createOffer("o1", "s1", mem, cpu)
scheduler.resourceOffers(driver, List(offer).asJava)

val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
val uris = launchedTasks.head.getCommand.getUrisList
assert(uris.stream().allMatch(_.getCache))
}

test("supports disabling fetcher cache") {
setScheduler(Map("spark.mesos.fetcherCache.enable" -> "false"))

val mem = 1000
val cpu = 1

val response = scheduler.submitDriver(
new MesosDriverDescription("d1", "jar", mem, cpu, true,
command,
Map("spark.mesos.executor.home" -> "test",
"spark.app.name" -> "test"),
"s1",
new Date()))

assert(response.success)

val offer = Utils.createOffer("o1", "s1", mem, cpu)
scheduler.resourceOffers(driver, List(offer).asJava)

val launchedTasks = Utils.verifyTaskLaunched(driver, "o1")
val uris = launchedTasks.head.getCommand.getUrisList
assert(uris.stream().allMatch(!_.getCache))
}

test("accept/decline offers with driver constraints") {
setScheduler()

0 comments on commit 3abf45d

Please sign in to comment.
You can’t perform that action at this time.
You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session.