[SPARK-25647][k8s] Add spark streaming compatibility suite for kubernetes. #22639

ScrapCodes · 2018-10-05T09:14:21Z

What changes were proposed in this pull request?

Adds integration tests for spark streaming compatibility with K8s mode.

How was this patch tested?

By running the test suites.

…etes.

ScrapCodes · 2018-10-05T09:18:40Z

...ts/src/test/scala/org/apache/spark/deploy/k8s/integrationtest/KubernetesTestComponents.scala

@@ -120,7 +120,7 @@ private[spark] object SparkAppLauncher extends Logging {
        appConf.toStringArray :+ appArguments.mainAppResource

    if (appArguments.appArgs.nonEmpty) {
-      commandLine += appArguments.appArgs.mkString(" ")
+      commandLine ++= appArguments.appArgs


Space separated single argument or, multiple different argument. If we do .mkString(" ") then, it takes multi arguments as space separated single argument.

SparkQA · 2018-10-05T09:28:24Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3709/

SparkQA · 2018-10-05T09:38:05Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3709/

SparkQA · 2018-10-05T09:39:20Z

Test build #96987 has finished for PR 22639 at commit d2230fc.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-05T10:13:30Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3711/

SparkQA · 2018-10-05T10:16:32Z

Test build #96989 has finished for PR 22639 at commit 29a8e2e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-05T10:25:53Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3711/

ScrapCodes · 2018-10-05T10:51:05Z

Jenkins, retest this please

SparkQA · 2018-10-05T11:04:00Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3713/

SparkQA · 2018-10-05T11:17:16Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3713/

SparkQA · 2018-10-05T11:18:10Z

Test build #96994 has finished for PR 22639 at commit 29a8e2e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

felixcheung · 2018-10-06T07:03:35Z

@liyinan926 @mccheah

mccheah

I think a lot of this could be cleaner if we deployed in cluster mode. Also deploying a separate pod to run the socket server with the extra connection for the word count to read from would be cleaner in terms of reasoning about resource management and not leaking sockets.

mccheah · 2018-10-08T22:49:20Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+
+  k8sSuite: KubernetesSuite =>
+
+  private def startSocketServer(): (String, Int, ServerSocket) = {


Is this the submission client starting a server, and then the pod needs to be able to connect to the submission client host and port? An alternative is to deploy a separate pod that does this so that network communications are pod-to-pod instead of pod-to-host.

If we do it that way it's a lot easier to clear the resources and avoid trouble with sockets hanging open on the Jenkins bare metal host, for example we can just delete the whole server pod.

mccheah · 2018-10-08T22:50:37Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+        logInfo(s"Received connection on $socket")
+        for (i <- 1 to 10 ) {
+          if (socket.isConnected && !serverSocket.isClosed) {
+            socket.getOutputStream.write("spark-streaming-kube test.\n".getBytes())


Specify encoding as UTF-8.

mccheah · 2018-10-08T22:54:58Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+    }
+  }
+
+  private def getRunLog(_driverPodName: String): String = kubernetesTestComponents.kubernetesClient


Why the underscore in _driverPodName? If it's a name conflict, I would presume the driver pod name is available elsewhere already and we can just use the local var and not have an argument.

mccheah · 2018-10-08T22:56:05Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+      .withImagePullPolicy("IfNotPresent")
+      .withCommand("/opt/spark/bin/run-example")
+      .addToArgs("--master", s"k8s://https://kubernetes.default.svc")
+      .addToArgs("--deploy-mode", mode)


If we're creating the pod manually, the deploy mode should always be client and not cluster, so this should just be hardcoded, and the argument removed from the function signature.

Also why are we running in client mode if we're going to be deploying a pod anyways? We can deploy this in cluster mode using spark submit and then set up the service in front of the pod anyways.

mccheah · 2018-10-08T22:56:59Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+
+  private def driverServiceSetup(_driverPodName: String): (Int, Int, Service) = {
+    val labels = Map("spark-app-selector" -> _driverPodName)
+    val driverPort = 7077


driverPort and blockManagerPort can be constants in a companion object.

ScrapCodes · 2018-10-11T10:41:04Z

@mccheah Thanks for taking a look. Overall nice suggestion, I am okay with idea of having a pod, but struggling with creating a pod for socket server. Can you please suggest how to go about it?

Not sure what should be the command for running, it.

SparkQA · 2018-10-11T11:18:36Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3879/

SparkQA · 2018-10-11T11:26:30Z

Test build #97250 has finished for PR 22639 at commit c63c54e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-11T11:32:28Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3879/

skonto · 2018-10-11T13:40:17Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+  import StreamingCompatibilitySuite._
+
+  test("Run spark streaming in client mode.", k8sTestTag) {
+    val (host, port, serverSocket) = startSocketServer()


We could use a custom source as an alternative for feeding the stream. Re-using existing code is also nice.

Please correct my understanding, a custom source has to either live in examples, or a separate image has to be published with the class path of the custom source.

I am more inclined to add in the examples. Just an alternative option.

ScrapCodes · 2018-10-15T09:57:16Z

Jenkins, retest this please

SparkQA · 2018-10-15T12:10:11Z

Test build #97381 has finished for PR 22639 at commit c63c54e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-15T12:10:25Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3975/

SparkQA · 2018-10-15T12:22:58Z

Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/3975/

ifilonenko · 2018-10-16T19:55:19Z

...src/test/scala/org/apache/spark/deploy/k8s/integrationtest/StreamingCompatibilitySuite.scala

+      .withLabels(labels.asJava)
+      .endMetadata()
+      .withNewSpec()
+      .withServiceAccountName(kubernetesTestComponents.serviceAccountName)


The flakiness of the tests seems to be in relation to this service account. Investigate the necessary rbac.yml that would need to be set to ensure that these failures don't come up?

The same is used in "run in client mode" test.

I am not sure, what is causing it. Do you have any clue?

I don't believe this was solved in the Client mode tests, despite the addition of the spark-rbac.yml I think this might require investigation outside of the scope of this PR.

I have run these tests on my own setup of minikube, and I am unable to reproduce the failure that occurred on jenkins. It is possible that is related to how minikube is setup on jenkins.

ScrapCodes · 2018-10-25T08:01:38Z

Jenkins, retest this please.

SparkQA · 2018-10-25T08:13:41Z

Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4468/

SparkQA · 2018-10-25T08:17:28Z

Test build #98006 has finished for PR 22639 at commit c63c54e.

This patch passes all tests.
This patch merges cleanly.
This patch adds no public classes.

SparkQA · 2018-10-25T08:26:56Z

Kubernetes integration test status success
URL: https://amplab.cs.berkeley.edu/jenkins/job/testing-k8s-prb-make-spark-distribution-unified/4468/

ScrapCodes · 2018-10-30T08:53:42Z

@mccheah and @skonto Do you have suggestion how to go forward from here? I wanted to write more tests, like how to recover from checkpoints etc...

skonto · 2019-01-15T14:56:00Z

@ScrapCodes could you update the PR according to @mccheah comments? I think we will be good to go then.

vanzin · 2019-03-08T18:34:23Z

If this isn't being worked on it should probably be closed.

ScrapCodes · 2019-03-09T07:34:01Z

I intend to finish this soon, if you like I can keep it closed until then?

[SPARK-25647][k8s] Add spark streaming compatibility suite for kubern…

d2230fc

…etes.

ScrapCodes force-pushed the stream-test-k8s branch from 0a4e222 to d2230fc Compare October 5, 2018 09:15

ScrapCodes commented Oct 5, 2018

View reviewed changes

Added to kubernetes suite to enable the test

29a8e2e

mccheah suggested changes Oct 8, 2018

View reviewed changes

ScrapCodes added 2 commits October 11, 2018 16:25

Trying to setup a pod for socket server.

729e5a9

Not sure what should be the command for running, it.

Refactored and incorporated code review.

c63c54e

skonto reviewed Oct 11, 2018

View reviewed changes

ifilonenko reviewed Oct 16, 2018

View reviewed changes

ScrapCodes closed this Mar 9, 2019


		k8sSuite: KubernetesSuite =>

		private def startSocketServer(): (String, Int, ServerSocket) = {

[SPARK-25647][k8s] Add spark streaming compatibility suite for kubernetes. #22639

[SPARK-25647][k8s] Add spark streaming compatibility suite for kubernetes. #22639

Conversation

ScrapCodes commented Oct 5, 2018

What changes were proposed in this pull request?

How was this patch tested?

Choose a reason for hiding this comment

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

ScrapCodes commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

SparkQA commented Oct 5, 2018

felixcheung commented Oct 6, 2018

mccheah left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScrapCodes commented Oct 11, 2018 • edited

SparkQA commented Oct 11, 2018

SparkQA commented Oct 11, 2018

SparkQA commented Oct 11, 2018

skonto Oct 11, 2018 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScrapCodes commented Oct 15, 2018

SparkQA commented Oct 15, 2018

SparkQA commented Oct 15, 2018

SparkQA commented Oct 15, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ScrapCodes commented Oct 25, 2018

SparkQA commented Oct 25, 2018

SparkQA commented Oct 25, 2018

SparkQA commented Oct 25, 2018

ScrapCodes commented Oct 30, 2018

skonto commented Jan 15, 2019

vanzin commented Mar 8, 2019

ScrapCodes commented Mar 9, 2019

ScrapCodes commented Oct 11, 2018 •

edited

skonto Oct 11, 2018 •

edited