Slurm Support

[ryandanehy]

Hi All,

I see slurm support was added recently and I am excited to take advantage of it. However, my slurm cluster configures GRES without a GPU type but with partitions. Looking through the skypilot documentation I see gpu_partition_map and I thought that would be my solution but it seems to not work? For those that have setup slurm infra with SkyPilot how have you mapped parititons, and if you willing could you share an example /sky/config.yaml?

Bonus Question: Has anyone tried to get skypilot to launch jobs via a Globus Compute Endpoint (I have a weird use case for it)?

[kevinmingtarja]

Hi @ryandanehy, yes that is exactly what gpu_partition_map is intended to solve, so you're on the right track!

Could you share more about what failures you're running into? Also could you share the output of sky api info?

For reference, this was what I tested with when developing the feature:

# ~/.sky/config.yaml
slurm:
  gpu_partition_map:
    H100: h100

And then sky launch --infra slurm --gpus H100:8 should route you to the h100 partition.

[ryandanehy]

Also some of the partitions have a shared queue and a exclusive queue. How does skypilot handle the difference or are the partitions treated the same?

[kevinmingtarja]

No worries, glad that it works now! And thanks for catching the bug on sky gpus list not using the gpu partition mapping.

is there a way to show the pricing of all my available infra?

Just for a specific GPU, or across all GPUs you have?

Also some of the partitions have a shared queue and a exclusive queue. How does skypilot handle the difference or are the partitions treated the same?

I believe we treat them the same today. We'd love to learn more about the use case though, sounds like something we can handle better.

Is shared vs exclusive queue basically OverSubscribe=YES/FORCE/NO vs OverSubscribe=EXCLUSIVE? Or is there any other config on the partition that differentiates these two.

[ryandanehy]

Yes that is correct. From a users perspective (I am an admin) shared means you can request one GPU and only be charged for that one GPU (other people are sharing your node) or you can request an exclusive node which you get all 8 GPUs and pay for all 8 GPUs regardless if you actually use them all.

Therefore it would be cool if skypilot could differentiate between the two so that if I request 1 h100 GPU it would queue up in the shared queue so I only get charged for the 1. Otherwise it will occasionally put me in the exclusive queues, and I would be charge for 8 even though I only use 1. Hopefully, that helps.

[ryandanehy]

Specific GPU or accross all GPUs either anwser would be interesting to me.

Further context for our institution:
Our institution has an on prem cluster, and we do alot of cloud computing (aws, gcp, azure). I was exploring skypilot as an option for our institute to adopt (Pacific Northwest National Lab) so that our users could submit jobs to both cloud and on prem resources from just their command line (as oppose to jumping between all the different reasources).

[kevinmingtarja]

Yes that is correct. From a users perspective (I am an admin) shared means you can request one GPU and only be charged for that one GPU (other people are sharing your node) or you can request an exclusive node which you get all 8 GPUs and pay for all 8 GPUs regardless if you actually use them all.

Therefore it would be cool if skypilot could differentiate between the two so that if I request 1 h100 GPU it would queue up in the shared queue so I only get charged for the 1. Otherwise it will occasionally put me in the exclusive queues, and I would be charge for 8 even though I only use 1. Hopefully, that helps.

Got it, I think this makes a lot of sense.

---

Specific GPU or accross all GPUs either anwser would be interesting to me.

For a specific GPU, you can already somewhat do this today using sky launch --gpus <GPU> and sky gpus list <GPU> (although it's a bit noisy today because we show the pricing for all clouds)

% sky launch --gpus H100:1 --dryrun
Considered resources (1 node):
-----------------------------------------------------------------------------------------------------
 INFRA                     INSTANCE                   vCPUs   Mem(GB)   GPUS     COST ($)   CHOSEN
-----------------------------------------------------------------------------------------------------
 Kubernetes (sky-dev-cw)   -                          4       16        H100:1   0.00          ✔
 Lambda (us-east-1)        gpu_1x_h100_pcie           26      200       H100:1   3.29
 GCP (us-central1-a)       a3-highgpu-1g              26      234       H100:1   5.38
 AWS (us-east-1)           p5.4xlarge                 16      256       H100:1   6.88
 Azure (eastus)            Standard_NC40ads_H100_v5   40      320       H100:1   6.98
-----------------------------------------------------------------------------------------------------
🔍 Multiple Lambda instances satisfy H100:1. The cheapest (gpus=H100:1, cpus=4, mem=16, ...) is considered among: gpu_1x_h100_pcie, gpu_1x_h100_sxm5.
To list more details, run: sky gpus list H100

% sky gpus list H100
Kubernetes GPUs
Context: sky-dev-cw
GPU   REQUESTABLE_QTY_PER_NODE  UTILIZATION
H100  1, 2, 4, 8                14 of 16 free 

No matching GPU nodes found in the Slurm cluster with filters (gpu_name='H100').
Resources 'H100' not found in any Slurm partition. To show available accelerators on Slurm, run: sky gpus list --cloud slurm

Cloud GPUs
GPU   QTY   CLOUD           INSTANCE_TYPE                             DEVICE_MEM  vCPUs  HOST_MEM  HOURLY_PRICE  HOURLY_SPOT_PRICE  REGION
...
H100  1.0   GCP             a3-highgpu-1g                             80GB        26     234GB     $ 5.383       $ 4.271            us-central1
H100  2.0   GCP             a3-highgpu-2g                             80GB        52     468GB     $ 10.767      $ 8.541            us-central1
H100  4.0   GCP             a3-highgpu-4g                             80GB        104    936GB     $ 21.533      $ 17.083           us-central1
H100  8.0   GCP             a3-highgpu-8g                             80GB        208    1872GB    $ 43.066      $ 34.165           us-central1
H100  1.0   AWS             p5.4xlarge                                80GB        16     256GB     $ 6.880       -                  us-east-1
H100  8.0   AWS             p5.48xlarge                               80GB        192    2048GB    $ 55.040      $ 10.295           us-east-2
H100  1.0   Azure           Standard_NC40ads_H100_v5                  -           40     320GB     $ 6.980       $ 6.980            eastus
H100  2.0   Azure           Standard_NC80adis_H100_v5                 -           80     640GB     $ 13.960      $ 13.960           eastus
H100  12.0  Azure           Standard_ND96isr_H100_v5                  -           96     1900GB    $ 98.320      $ 98.320           eastus
...

[ryandanehy]

Thank you Kevin for all your help. Some last questions:

• If the slurm queue is full does skypilot revert to using the next cheapest available resource?
• Have you seen anyone use a workflow using skypilot that runs individual tasks on different cloud providers and then combine the results (a true hybrid workflow). Example: Run part of an application on a slurm based infra and at the same time run part of the application in the cloud.

Also I appreciate your help. If you need any help testing new Slurm features or want help brainstorming ideas let me know.

[kevinmingtarja]

If the slurm queue is full does skypilot revert to using the next cheapest available resource?

Yes, SkyPilot will fail over to other resources (other Slurm partitions, other Slurm clusters, or other clouds as ranked by the optimizer) when the Slurm queue can't allocate nodes for the job in time. The mechanism is the slurm.provision_timeout config in ~/.sky/config.yaml:

slurm:
  provision_timeout: 120  # seconds
  cluster_configs:
    mycluster:
      provision_timeout: 600

Defaults:

• No partition specified → 120s, so failover kicks in quickly.
• Partition pinned (e.g. --infra slurm/mycluster/mypartition) → 24h, since failover is disabled anyway and we just let Slurm queue the job.

If a job stays PENDING past the timeout, SkyPilot will scancel the current job in the queue, and move on to the next candidate (next partition → next Slurm cluster → next cloud). So you can play around with provision_timeout to tune the behaviour.

---

Have you seen anyone use a workflow using skypilot that runs individual tasks on different cloud providers and then combine the results

Yes absolutely, basic sequential pipelines work natively. Each task in a multi-task YAML has its own resources block, so you can pin task 1 to Slurm, task 2 to AWS, task 3 to GCP, and pass data between them via shared storage (S3, GCS, etc). See Managed pipelines for what the YAML looks like.

Anything beyond a sequential pipeline (branching, fan-in/fan-out, "run Slurm and AWS parts concurrently then merge") isn't expressible in our YAML today. For that, people usually use another DAG/workflow engine on top like Airflow and Temporal, which then wraps individual SkyPilot launches as steps in the DAG. We have examples on how to integrate with those here and here.

[kevinmingtarja]

Also I appreciate your help. If you need any help testing new Slurm features or want help brainstorming ideas let me know.

Of course, happy to help! And yes thank you for the offer, we definitely need all the help we can get in terms of testing on different Slurm clusters and environments.

[ryandanehy]

Hi @kevinmingtarja,

My team and I would like to schedule a meeting with you and maybe some others on the Skypilot team. We want to have an open ended discussion around some of our Continuum Computing work we are doing and how Skypilot might fit in. Would the team be interested in this?

[kevinmingtarja]

That sounds great, we'd definitely be interested, should we continue the conversation on slack?