[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases #17844

cloud-fan · 2017-05-03T18:09:38Z

What changes were proposed in this pull request?

ReplSuite.newProductSeqEncoder with REPL defined class was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of ReplSuite, we create a REPL instance, which creates a classloader and loads a lot of classes related to SparkContext. More details please see #17833 (comment).

In this PR, we create a new test suite, SingletonReplSuite, which shares one REPL instances among all the test cases. Then we move most of the tests from ReplSuite to SingletonReplSuite, to avoid creating a lot of REPL instances and reduce memory footprint.

How was this patch tested?

test only change

cloud-fan · 2017-05-03T18:09:50Z

cc @zsxwing @sameeragarwal

SparkQA · 2017-05-03T20:13:30Z

Test build #76423 has finished for PR 17844 at commit 1565a34.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

sameeragarwal · 2017-05-04T18:13:24Z

test this please

SparkQA · 2017-05-04T20:22:04Z

Test build #76461 has finished for PR 17844 at commit 1565a34.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-05T05:20:27Z

Test build #76471 has finished for PR 17844 at commit 9248a5e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

zsxwing

Looks pretty good. Just some nits.

zsxwing · 2017-05-05T18:28:08Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala

+    val currentOffset = out.getBuffer.length()
+    // append a `val _result = 1` statement to the end of the given code, so that we can know what's
+    // the final output of this code snippet and rely on it to wait until the output is ready.
+    in.write((input + s"\nval _result = 1\n").getBytes)


nit: it's better to use some more special string. E.g.,

val timestamp = System.currentTimeMillis() in.write((input + s"\nval _result_$timestamp = 1\n").getBytes) in.flush() val stopMessage = s"_result_$timestamp: Int = 1"

zsxwing · 2017-05-05T22:06:04Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala

+    }
+  }
+
+  def runInterpreter(input: String): String = {


Could you add a comment mentioning that the codes in input should not throw an exception?

zsxwing · 2017-05-05T22:09:26Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala

+    super.afterAll()
+  }
+
+  private def waitUntil(cond: () => Boolean): Unit = {


You can use eventually

eventually(timeout(50.seconds), interval(500.millis)) { assert(cond(), "current output: " + out.toString) }

zsxwing · 2017-05-05T22:22:22Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/ReplSuite.scala

-    assertDoesNotContain("Exception", output)
-    assertContains(": Int = 20", output)
-  }
-
  test("line wrapper only initialized once when used as encoder outer scope") {


Why you don't move this test?

this test only works in local mode...

zsxwing · 2017-05-05T22:25:49Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala

+    val classpath = paths.map(new File(_).getAbsolutePath).mkString(File.pathSeparator)
+
+    System.setProperty(CONF_EXECUTOR_CLASSPATH, classpath)
+    Main.conf.set("spark.master", "local-cluster[2,4,4096]")


nit: I think it's safe to just use local-cluster[2,1,1024].

zsxwing · 2017-05-05T22:27:53Z

repl/scala-2.11/src/test/scala/org/apache/spark/repl/SingletonReplSuite.scala

+    Main.sparkSession = null
+
+    // Starts a new thread to run the REPL interpreter, so that we won't block.
+    thread = new Thread(new Runnable {


nit: please call Thread.setDaemon(true) in case that something is wrong and it prevents the process from exiting.

SparkQA · 2017-05-08T13:02:37Z

Test build #76569 has finished for PR 17844 at commit fd5b43b.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-08T13:22:28Z

retest this please

SparkQA · 2017-05-08T16:17:51Z

Test build #76577 has finished for PR 17844 at commit fd5b43b.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-08T16:44:47Z

retest this please

SparkQA · 2017-05-08T18:47:55Z

Test build #76582 has finished for PR 17844 at commit fd5b43b.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-09T00:16:12Z

retest this please

SparkQA · 2017-05-09T03:06:44Z

Test build #76600 has finished for PR 17844 at commit fd5b43b.

This patch fails SparkR unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-09T04:12:37Z

Test build #76618 has started for PR 17844 at commit 7dde745.

gatorsmile · 2017-05-09T07:07:08Z

retest this please

SparkQA · 2017-05-09T09:23:34Z

Test build #76649 has finished for PR 17844 at commit 7dde745.

This patch fails Spark unit tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2017-05-09T11:22:45Z

Test build #76656 has finished for PR 17844 at commit 7dde745.

This patch fails from timeout after a configured wait of `250m`.
This patch merges cleanly.
This patch adds no public classes.

cloud-fan · 2017-05-09T12:12:38Z

retest this please

SparkQA · 2017-05-09T15:04:19Z

Test build #76675 has finished for PR 17844 at commit 7dde745.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

`ReplSuite.newProductSeqEncoder with REPL defined class` was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of `ReplSuite`, we create a REPL instance, which creates a classloader and loads a lot of classes related to `SparkContext`. More details please see #17833 (comment). In this PR, we create a new test suite, `SingletonReplSuite`, which shares one REPL instances among all the test cases. Then we move most of the tests from `ReplSuite` to `SingletonReplSuite`, to avoid creating a lot of REPL instances and reduce memory footprint. test only change Author: Wenchen Fan <wenchen@databricks.com> Closes #17844 from cloud-fan/flaky-test. (cherry picked from commit f561a76) Signed-off-by: Wenchen Fan <wenchen@databricks.com>

cloud-fan · 2017-05-09T16:12:10Z

thanks for the review, merging to maser/2.2!

## What changes were proposed in this pull request? `ReplSuite.newProductSeqEncoder with REPL defined class` was flaky and throws OOM exception frequently. By analyzing the heap dump, we found the reason is that, in each test case of `ReplSuite`, we create a REPL instance, which creates a classloader and loads a lot of classes related to `SparkContext`. More details please see apache#17833 (comment). In this PR, we create a new test suite, `SingletonReplSuite`, which shares one REPL instances among all the test cases. Then we move most of the tests from `ReplSuite` to `SingletonReplSuite`, to avoid creating a lot of REPL instances and reduce memory footprint. ## How was this patch tested? test only change Author: Wenchen Fan <wenchen@databricks.com> Closes apache#17844 from cloud-fan/flaky-test.

cloud-fan force-pushed the flaky-test branch from 1565a34 to 9248a5e Compare May 5, 2017 02:32

zsxwing reviewed May 5, 2017

View reviewed changes

cloud-fan added 2 commits May 9, 2017 12:10

share one REPL instance for REPL test cases

3d14998

address comments

7dde745

cloud-fan force-pushed the flaky-test branch from fd5b43b to 7dde745 Compare May 9, 2017 04:10

asfgit closed this in f561a76 May 9, 2017

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases #17844

[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases #17844

cloud-fan commented May 3, 2017

cloud-fan commented May 3, 2017

SparkQA commented May 3, 2017

sameeragarwal commented May 4, 2017

SparkQA commented May 4, 2017

SparkQA commented May 5, 2017

zsxwing left a comment

zsxwing May 5, 2017

zsxwing May 5, 2017

zsxwing May 5, 2017

zsxwing May 5, 2017

cloud-fan May 8, 2017

zsxwing May 5, 2017

zsxwing May 5, 2017

SparkQA commented May 8, 2017

cloud-fan commented May 8, 2017

SparkQA commented May 8, 2017

cloud-fan commented May 8, 2017

SparkQA commented May 8, 2017

cloud-fan commented May 9, 2017

SparkQA commented May 9, 2017

SparkQA commented May 9, 2017

gatorsmile commented May 9, 2017

SparkQA commented May 9, 2017

SparkQA commented May 9, 2017

cloud-fan commented May 9, 2017

SparkQA commented May 9, 2017

cloud-fan commented May 9, 2017

[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases #17844

[SPARK-20548][FLAKY-TEST] share one REPL instance among REPL test cases #17844

Conversation

cloud-fan commented May 3, 2017

What changes were proposed in this pull request?

How was this patch tested?

cloud-fan commented May 3, 2017

SparkQA commented May 3, 2017

sameeragarwal commented May 4, 2017

SparkQA commented May 4, 2017

SparkQA commented May 5, 2017

zsxwing left a comment

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

cloud-fan May 8, 2017

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

zsxwing May 5, 2017

Choose a reason for hiding this comment

SparkQA commented May 8, 2017

cloud-fan commented May 8, 2017

SparkQA commented May 8, 2017

cloud-fan commented May 8, 2017

SparkQA commented May 8, 2017

cloud-fan commented May 9, 2017

SparkQA commented May 9, 2017

SparkQA commented May 9, 2017

gatorsmile commented May 9, 2017

SparkQA commented May 9, 2017

SparkQA commented May 9, 2017

cloud-fan commented May 9, 2017

SparkQA commented May 9, 2017

cloud-fan commented May 9, 2017