Skip to content

Conversation

@dongjoon-hyun
Copy link
Member

@dongjoon-hyun dongjoon-hyun commented Mar 4, 2025

What changes were proposed in this pull request?

This PR aims to add submit-spark-connect-server.sh example to run SparkConnectServer on Spark Standalone Cluster.

Note that there are other ways to launch SparkConnectServer and this only aims to focus on Spark Standalone case.

Why are the changes needed?

  1. Install Nightly Spark Kubernetes Operator and run a Spark cluster.
$ helm install spark-kubernetes-operator \
https://nightlies.apache.org/spark/charts/spark-kubernetes-operator-0.1.0-SNAPSHOT.tgz
NAME: spark-kubernetes-operator
LAST DEPLOYED: Mon Mar  3 20:55:41 2025
NAMESPACE: default
STATUS: deployed
REVISION: 1

$ kubectl apply -f prod-cluster-with-three-workers.yaml
sparkcluster.spark.apache.org/prod created
  1. Setup port-forwarding.
$ kubectl port-forward svc/prod-master-svc 6066:6066
$ kubectl port-forward svc/prod-master-svc 6066:6066
  1. Submit SparkConnectServer.
$ ./submit-spark-connect-server.sh
{
  "action" : "CreateSubmissionResponse",
  "message" : "Driver successfully submitted as driver-20250304045839-0000",
  "serverSparkVersion" : "4.0.0-preview2",
  "submissionId" : "driver-20250304045839-0000",
  "success" : true
}%
Screenshot 2025-03-03 at 21 07 06 Screenshot 2025-03-03 at 21 05 22

Does this PR introduce any user-facing change?

No. This is a newly added example.

How was this patch tested?

Manually follow the procedure.

Was this patch authored or co-authored using generative AI tooling?

No.

@dongjoon-hyun
Copy link
Member Author

Thank you, @viirya .

@dongjoon-hyun
Copy link
Member Author

Merged to main.

@dongjoon-hyun dongjoon-hyun deleted the SPARK-51382 branch March 4, 2025 05:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants