calculating resources with clusterscope

[luccabb]

Why ?

clusterscope has an API to calculate proportionate resource usage in the machine (CPUs / Mem) given the number of GPUs or CPUs.

notice how the original code has the following args:

"ntasks_per_node": 1,
"cpus_per_task": 96,
"gpus_per_node": 8,

H100 nodes have 192 cpus, the code above only uses 96 cpus per node (96 cpus_per_task * 1 ntasks_per_node), leaving the other 96 cpus idle.

what clusterscope provides is an API that will allocate the appropriate number of CPUs and Memory given the number of GPUs requested. it goes something like:

1. request 8 GPUs
2. clusterscope gets total cpus and memory for the nodes
3. clusterscope allocates cpus/memory based on the proportion of GPUs being requested

H100 nodes have 8 GPUs total, so this means clusterscope will allocate all memory and all cpus

Test plan

github ci

[skalyan]

Looks okay to me, unless there are some special case considerations for asking hard-coded CPU core count

[dongwang218]

maybe, we can also retire https://github.com/facebookresearch/matrix/blob/main/matrix/cluster/ray_worker_job.py#L30-L43?

[luccabb]

maybe, we can also retire https://github.com/facebookresearch/matrix/blob/main/matrix/cluster/ray_worker_job.py#L30-L43?

yeah I think we can retire this. I'll have to write code on the clusterscope side to support this operation, so I think its best to leave it for a future PR

[dongwang218]

sure, we can do in a follow up.

maybe, we can also retire https://github.com/facebookresearch/matrix/blob/main/matrix/cluster/ray_worker_job.py#L30-L43?

yeah I think we can retire this. I'll have to write code on the clusterscope side to support this operation, so I think its best to leave it for a future PR

[skalyan]

@dongwang218 where do you set this env variable? is it custom to you?

num_gpus = int(os.environ.get("SLURM_GPUS_ON_NODE", 0))