First Preview of Kluster - DASK Clusters in Arkitekt

[jhnnsrs]

Introducing Kluster

As stated in the Documentation Arkitekt aims not to be a replacement for existing highly parlellized computing platforms like the amazing Dask, but wants to provide more usable interfaces to interact with them.

With Kluster we provide a first way to interface with more dedicated high performance computing platforms. Building on the amazing Dask Gateway, Kluster is a WIP of the integration of Dask Clusters into Arkitekt.

Features

• Completely Integrated Cluster Management with JWT Authentication

• Manage Dask Clusters on a per User/App basis

  1. Directly interface with them through Orkestrator
  2. Scale, Adapt and Manage Resources

• Connect to SLURM and PBS (and potentially Kubernetes) (highly experimental)

• Use the familiar Args, Kwargs Syntax to allow apps to accept a Dask Cluster

  from kluster.api.schema import DaskClusterFragment
  
  def ineed_a_cluster(cluster: DaskClusterFragment) -> int:
       gateway = cluster.get_gateway()
       client = gateway.get_client()
       # do stuff with your dask distribued client

• "Zero" Configuration and integratable into standard Arkiekt Deployment

Video

Here's a very early version of it in action.

cluster_yes.webm

Roadmap

• [ x] Initial Prototype
• Better User Interface
• Test integration with SLURM cluster
• Streamline Test Environment
• Your Feedback

[jhnnsrs]

@alexschroeter

[alexschroeter]

Ok sorry for the long delay but I had to do some stuff and collect my thoughts. In my head, it boils down to two parts.

Integration of computing resources into Arkitekt workloads

Everything that is sort of based on cloud is not an issue because they provide the tools to get access and if they want to make money they need to keep doing so. HPC is a different beast in the way they provide access, sometimes for good reasons and sometimes for bad reasons.

I am a bit fearful that the dask-gateway-server will be a hindrance for many people because the HPC center or maybe the local admins of their compute infrastructure won't be able to set this up and the dask-gateway-server needs admin support.

This is my main reason for looking into flux which if I understand correctly, you would start via the resource manager provided by your compute infrastructure (slurm, Kubernetes, some script, ...) and join the instance running on arkitekt. This way the resources would be managed by Arkitekt where users could add resources dynamically and of course start things like Dask if they are needed for a big job.

My fear with this approach is that it gets too complicated and we would be dependent on another open source tool that might not stand the test of time.

Dask itself

What you have created in such a short time looks great. We have to figure out a couple of things like:

• Dask Distributed has an Nvidia/CUDA-specific setup and I am wondering what needs to be done there to support AMD/ROCM stuff

Roadmap

• [ x] Initial Prototype
  Looks Amazing
• Better User Interface
  What sort of interface is needed here? I believe the visualization of the compute resources you already have. A question would be how to handle if you have more computing resources than you need. Dask I believe handles the distribution of jobs so maybe some interface to the cost function or blocking certain resources temporarily. I don't know.
• Test integration with SLURM cluster
  This is very much dependent on my previous statement. I believe if we want to go the route of the dask-gateway-server, I have to get this started either in a test environment or try to get this rolled out on the cluster.
• Streamline Test Environment
  I would skip this until we have a more concrete plan.
• Your Feedback
  These are my thoughts. When you are done reading them I suggest we have a call where we talk them out. I am also not sure if I am overcomplicating things but my gut says the dask-gateway-server will be a showstopper for many.