[Question] Custom criteria for select_devices

[ioslide]

Required prerequisites

• I have searched the Issue Tracker that this hasn't already been reported. (comment there if it has.)
• I have tried the latest version of nvitop in a new isolated virtual environment.

Motivation

Can "select_devices" be updated to specify a GPU and policy? If the policy is set to "queue", the specified GPU can be held until it meets the requirements.

Solution

No response

Alternatives

No response

Additional context

No response

[XuehaiPan]

Can "select_devices" be updated to specify a GPU and policy?

You can pass devices argument to select_devices. Maybe also add sort=False.

devices = select_devices(devices, <criteria args>, sort=False)

For nvisel CLI, use CUDA_VIRIBLE_DEVICES and --no-sort (-S):

$ nvisel -i "GPU-18ef14e9,GPU-849d5a8d" -S [<criteria options>]
5,6

---

If the policy is set to "queue", the specified GPU can be held until it meets the requirements.

You can write your own logic about that. It may be too complex to add in nvitop.select natively.

[XuehaiPan]

Closing for it is a specific usage and really depends on the hardware setup. Workload management is kind of out of topic. You can always write your own logic for device selection or use specialized software like slurm.