Apache Spark 2.3 app on which Spark Operator version

[Fixmetal]

Hello everybody,
I've setup this operator to be working for a bunch of applications. I've been using v1beta2-1.1.2-2.4.5.
I'm struggling because I don't understand the compatibility between Spark and the Spark operator really. The overview states Supports Spark 2.3 and up. so I've come to the idea that for running an Apache Spark 2.3 application I've got to define the SparkApplication CR like:

spec.sparkVersion: 2.3.0

Does this suffice? I use the gcr.io/spark-operator/spark:v2.4.5 as a Base Image.
Thanks

[liyinan926]

You will need a spark operator version based on Spark 2.3.x, like this one https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/releases/tag/v1alpha1-0.4-2.3.x. We stopped maintaining the 2.3 based branch a long time ago so it's lacking features.

[Fixmetal]

wouldn't the multi-version-support branch work? isn't it purposed for supporting both 2.3 and 2.4?

[liyinan926]

Sure, you can use that branch.

[Fixmetal]

Sure, you can use that branch.

Well in the meantime I started using that. I suppose to make a 2.3 app work it is correct to just override the spec.sparkVersion and spec.driver.labels.version spec.executor.labels.version, isn't it?

[liyinan926]

Yes, and there's some API difference here. You can specify the k8s service account to be used by the pod running spark-submit using the following field:

spec:
  serviceAccount: <your sa>

[Fixmetal]

Thanks for confirming that. Seems I did everything correctly. Thank you very much.
Now I'm facing some OOM waddayaknow-whatever errors and can't get any ideas on why is that but that's a whole different story

[Fixmetal]

@liyinan926 : just to shed some light on this: which base image version should an application based on 2.3 be using with multi-support-version? Is gcr.io/spark-operator/spark:v2.4.5-gcs-prometheus fine or 2.3 specific applications should rely on previous gcr.io/spark-operator/spark:v2.3.0

[liyinan926]

gcr.io/spark-operator/spark:v2.3.0 is the one to use. You can also build your own 2.3.x image if you want to have custom jars/deps.

[Fixmetal]

gcr.io/spark-operator/spark:v2.3.0 is the one to use. You can also build your own 2.3.x image if you want to have custom jars/deps.

Thanks a lot. It happens to be working with 2.4.5 as base image really.
Might I suggest to add all these infos in the multi-support-version branch docs?

[Fixmetal]

Hello there,
I just patched my 2.3.x applications to use gcr.io/spark-operator/spark:v2.3.1-gcs-prometheus as a base image, fixing the #591 by using version 4.6.4 of kubernetes-client and fixing dependencies following this since I'm using K8S v1.15.3 - I've tested 4.7.1 in case of future K8S upgrade compatibility matrix but it doesn't work at all.

I patched the SparkApplications with 2.3.1:

$ yq r app.yaml  spec.sparkVersion
2.3.1
$ yq r app.yaml  spec.driver.labels
version: 2.3.1
$ yq r app.yaml spec.executor.labels
version: 2.3.1

Now I'm facing this exact same issue #564:

+ exec /sbin/tini -s -- /usr/lib/jvm/java-1.8-openjdk/bin/java -cp ':/opt/spark/jars/*' -Xms -Xmx -Dspark.driver.bindAddress=10.233.71.148
Invalid initial heap size: -Xms
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.

I'm using the multi-version-support branch and the operator is using its original image: gcr.io/spark-operator/spark-operator:v2.4.5-v1beta2-latest just as relative manifest

So now I'm kind of at a loss now because:

• you suggested to use the 2.3 image, but it doesn't work
• using gcr.io/spark-operator/spark:v2.4.5-gcs-prometheus as a base image and specificing sparkVersion: 2.3.1 they work even if the Spark Console shows 2.4.5 at top

Would you mind help sorting this out?

[Fixmetal]

Any news on this?

[Fixmetal]

bump. no support at all I changed up my solution.