Job instead of deployment for ephemeral Spark Clusters

[4n4nd]

I think instead of a deployment on openshift, a job would be a better type for ephemeral spark clusters as after the application is finished, it would terminate and not occupy resources and also it would be easier to schedule them as cronjobs.
Currently if I create a job, it does not create an ephemeral spark cluster, instead it creates a shared cluster which is not deleted when the job is finished.

[elmiko]

i think this is a good request, although we probably will need to support both styles of application: Deployment and Job.

i wonder if there is some way we can get the tooling to pick up either a Deployment or Job for the ephemeral cluster management?

also, i imagine this will get much easier to manage depending on how the CRD-based implementations work out.

[4n4nd]

A simple work-around for this: https://github.com/4n4nd/oc_train_pipeline/blob/master/delete_spark.sh as suggested by elmiko.
Just run this script after you are done with your spark cluster (I run it after sc.stop()) and it will forcefully delete it, so name your cluster carefully if you're naming it manually.