KFP version: 1.7.0+ Kubernetes version: 1.17+

Orchestrate Spark Jobs using Kubeflow pipelines

Install kubeflow pipelines standalone or full kubeflow

for standalone kubeflow pipelines installation

https://www.kubeflow.org/docs/components/pipelines/installation/

for full kubeflow installation

https://www.kubeflow.org/docs/started/installing-kubeflow/

Install Spark Operator

https://github.com/GoogleCloudPlatform/spark-on-k8s-operator#installation

Create Spark Service Account and add permissions

kubectl apply -f ./scripts/spark-rbac.yaml

Run the notebok kubeflow-pipeline.ipynb

Access Kubflow/KFP UI

OR

Upload pipeline

Upload the spark_job_pipeline.yaml file

Create Run

Start Pipeline add service account spark-sa

Wait till the execution is finished. check the print-message logs to view the result