Action items for adoption of Spark Kubernetes Operator in Kubeflow

[andreyvelich]

Related: kubeflow/community#648, kubeflow/community#672, #1892

We have to follow these action items to transfer Spark Operator to Kubeflow.

• Get approval from Kubeflow Steering Committee to accept Spark Operator as a Kubeflow core component.
• Transfer https://github.com/GoogleCloudPlatform/spark-on-k8s-operator repository to https://github.com/kubeflow/spark-operator.
• Configure tide for the new repository.
• Add Spark Operator maintainers: @vara-bonthu @yuchaoran2011 @mwielgus to the Kubeflow Org: https://github.com/kubeflow/internal-acls?tab=readme-ov-file#joining-kubeflow-github-organization
• Create OWNERS file with the maintainers: @vara-bonthu @yuchaoran2011 @mwielgus in the new Kubeflow repository.
• Modify the contributing doc to follow Kubeflow contributing guidelines (e.g. licence agreement if that is required).
• Attend to repo migration side effect in 404 Not Found https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml  #1926.
• Update docs with new helm charts link. (For example, https://github.com/kubeflow/spark-operator/tree/master?tab=readme-ov-file#installation)
• Fix GitHub actions for helm charts.
• Add Docker Hub Token to Spark Operator repo
• Fix GitHub actions for container registry (Kubeflow DockerHub).
• Migrate Spark Operator docs to Kubeflow Website..
• Improve the release process: Improve Spark Operator release process #1975

Please let me know if I missed something @jbottum @james-jwu @zijianjoy.
Excited to see this moving forward 🎉

[jbottum]

this looks complete, let's move this forward

[zijianjoy]

To share more information about repo config and tide config: We will follow the instruction in https://github.com/kubeflow/community/blob/master/repository-setup.md#repository-configuration to set up this new repo. Feel free to let me know if you need my help on the admin side of steps.

[andreyvelich]

Great, thank you, James!

[nttq1sub]

@andreyvelich any update on this

[andreyvelich]

@andreyvelich any update on this

We are working with @zijianjoy and Google team to transfer Spark Operator repository to Kubeflow GitHub.
I will share updates when we have more information.
Feel free to join Kubeflow Community Call today to hear about the latest updates.

[vikas-saxena02]

@andreyvelich do we know when will the work start on this?

[andreyvelich]

@andreyvelich do we know when will the work start on this?

Once we transfer the Spark Operator repository to Kubeflow the team will start work on the items to make a release.
In the meantime, it would be nice if you @vikas-saxena02 could help to answer some user questions in the #kubeflow-spark-operator Slack Channel: https://kubeflow.slack.com/archives/C06627U3XU3

[gangahiremath]

I am not able to sign in to kubeflow slack. Please share the steps. Thank you

[andreyvelich]

I am not able to sign in to kubeflow slack. Please share the steps. Thank you

@gangahiremath Please follow this link to join Kubeflow Slack: https://www.kubeflow.org/docs/about/community/#kubeflow-slack

[vikas-saxena02]

sure @andreyvelich
Quick question is there work planned to run spark jobs from kubeflow notebooks (jupyter notebooks)?

[andreyvelich]

sure @andreyvelich Quick question is there work planned to run spark jobs from kubeflow notebooks (jupyter notebooks)?

We have this item in our design doc: https://docs.google.com/document/d/1rCPEBQZPKnk0m7kcA5aHPf0fISl0MTAzsa4Wg3dfs5M/edit#heading=h.sb3itb5lb4jl.
So if you want to start this effort that would be nice.

[zijianjoy]

The repo is migrated and renamed to https://github.com/kubeflow/spark-operator.

[zijianjoy]

This link is helpful for setting up tide and prow: https://github.com/kubeflow/community/blob/master/repository-setup.md#setup-prow

[sid-habu]

With this move what's the name of the helm repo? The existing repo doesn't work

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator

Error: looks like "https://googlecloudplatform.github.io/spark-on-k8s-operator" is not a valid chart repository or cannot be reached: failed to fetch https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml : 404 Not Found

[sid-habu]

With this move what's the name of the helm repo? The existing repo doesn't work

helm repo add spark-operator https://googlecloudplatform.github.io/spark-on-k8s-operator

Error: looks like "https://googlecloudplatform.github.io/spark-on-k8s-operator" is not a valid chart repository or cannot be reached: failed to fetch https://googlecloudplatform.github.io/spark-on-k8s-operator/index.yaml : 404 Not Found

Nvm, I found the new repo link

helm repo add spark-operator https://kubeflow.github.io/spark-operator/