shell: support `-o gpu-affinity=map:LIST` #5356

Problem: We'd like to support a map:LIST option for the gpu affinity shell plugin that matches the cpu affinity support, but the functions to create, parse, and destroy lists of hwloc_cpuset_t objects are internal to the shell CPU affinity plugin. Export the cpuset array functions from affinity.c for use by other builtin shell plugins.

Problem: The gpubind plugin "per-task" works by allocating gpu ids from a shared gpus idset within the task.init callback, but this is different from how the cpu affinity plugins and thus code can't be easily shared. Rewrite the gpubind plugin to use an array of hwloc_cpuset_t objects to maintain which gpus should be assinged to each task. To ease memory management, add a context object that contains relevant data for the plugin, and create this object at initialization. The gpubind plugin now assigns GPUs to tasks during initialization, as does the affinity plugin. This allows the cpuset array code to be reused, and for the gpubind plugin to support a "map:LIST" option in the future.

Problem: The usefulness of the `-o cpu-affinity=map:LIST` option is reduced because the assigned cores must be contained within the cpu set assigned to the job. However, if a user is using the `map:LIST` option then they are presumably agreeing to take complete control of core assignments, and the option should not limit them to only the assigned cores. Drop the check in the affinity plugin that cores are contained within the job cpuset. Don't even check if the cores are valid (this would presumably fail later when the affinity is actually applied.) Fixes flux-framework#5352

Problem: Conditional registration of the task.init callback may cause code duplication if the callback is required in other situations. Always register the task.init callback. Do nothing if ctx->gpusets has not been assigned. This means there are no per-task GPUs.

Problem: The gpubind plugin returns early if no GPUs were allocated to the job, but some future gpu-affinity options may want to override the default binding in this case. Let the plugin fall through to the if/else block even if no GPUs are assigned to the job. Change the final `else` to `else if (ngpus > 0)` though, so that the plugin still does nothing (besides setting CUDA_VISIBLE_DEVICES=-1) if ngpus == 0.

Problem: The gpubind shell plugin doesn't support explicit mapping of GPUs to tasks. Support a `-o gpu-affinity=map:LIST` which works similar to the cpu-affinity of the same name. This option allows explicity specification of GPUs to tasks without regard for the actual GPU ids assigned to a job. It is mainly meant for testing, benchmarks, or when default GPU assignment is not working for a particular situation. Fixes flux-framework#5350

Problem: No tests in the testsuite ensure proper operation of the job shell gpu-affinity=map: option. Add a simple test that ensures the gpu-affinity `map:` option is working for a basic scenario.

Problem: The `map:` argument of the `gpu-affinity` shell option is not documented. Add a short description of this option to flux-shell(1).

Problem: The documentation of the cpu-affinity shell option in the flux-shell(1) manpage references the `hwloc_cpuset_t(3)` manpage, but the link does not exist. Reference the `hwlocality_bitmap(3)` manpage instead, and use a direct link to the hwloc docs v2.9.0 since hwlocality_bitmap(3) doesn't exist on the default manpage site (linux.die.net). Update the spelling dictionary.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

shell: support `-o gpu-affinity=map:LIST` #5356

shell: support `-o gpu-affinity=map:LIST` #5356

Commits on Jul 28, 2023

shell: support -o gpu-affinity=map:LIST #5356

shell: support -o gpu-affinity=map:LIST #5356

Commits on Jul 28, 2023

shell: support `-o gpu-affinity=map:LIST` #5356

shell: support `-o gpu-affinity=map:LIST` #5356