Batch Orchestration

[vsoch]

This is a discussion thread for the Batch Orchestration idea. You can:

• vote this discussion up to express interest in the idea
• add comments, feedback, or ideas for discussion in this thread.

[sk8forether]

Should we bring Kubernetes into this discussion? I have seen more papers about running Kubernetes with various methods of batch orchestration on HPC of late (one example) as a possible solution. There are also connections to bursting to cloud that could come into play when discussing possible solutions. If one desires interoperability between HPC cluster nodes with a virtual cluster deployed in the cloud that can be seamlessly deployed to as well, then Kubernetes could come into the discussion. Perhaps the most relevant angle to consider is how any suggested scheduler changes would work with Kubernetes as well, whether in the cloud or on an HPC resource?

[vsoch]

I think that would be good to add. I was just thinking about Kubernetes from our label/metadata discussion! I would be interested to know how many hybrid setups actually exist - we did have a Kubernetes cluster at Stanford but it was very separate from HPC.

[sk8forether]

This also seems like a good link to add somewhere, as one example of a scheduler and how it currently works with various container technologies: https://slurm.schedmd.com/containers.html

[vsoch]

That's a great idea! Do you want to propose a PR to add it?