NUMA setting and affinity configuration?

[biocyberman]

One of our software (CST) requires that the job running it must have correct NUMA setting. I don't really know what it is talking about. But here is SLURM documentation about cpu-bind and NUMA:
https://slurm.schedmd.com/mc_support.html#srun_lowlevelmc

Since SLURM documention was not helpful on this right now, and I forgot where I read about the argument for task affinity, I am asking it here:

1. Why affinity is needed?
2. What is the consequence (i.e. resource utility, runtime, etc) if I disable this plugin?
3. According to SLURM man page: man srun, some cpu-bind modes only support if the whole node is allocated. This is very unlikely case, provided how fat a DGX2 node is. So, what if I can have correct NUMA setting or the similar while I allocate a node partially?

[twittidai]

Just my 5 cents:

Why affinity is needed?

HPC applications (or any other applications that is neatly tuned for performance) typically utilize the underlying hardware architecture maximize performance. With task/cpu/gpu affinity, it allow the task to stay always on the same core/gpu, and have the minimum latency to data (or in some cases, reduce the amount data that needs to be copied), as data is typically allocated on that part of the memory which is close to the core calling the allocation (google keyword: first-touch).

What is the consequence (i.e. resource utility, runtime, etc) if I disable this plugin?

That depends on how well the application can utilize (benefit) from cpu/gpu affinity. In general, decreased performance / increased runtime is expected. Always remember: data access is very very time consuming and expensive.

According to SLURM man page: man srun, some cpu-bind modes only support if the whole node is allocated. This is very unlikely case, provided how fat a DGX2 node is. So, what if I can have correct NUMA setting or the similar while I allocate a node partially?

I don't have enough knowledge of the CST itself, but i found out it's an application based on MPI.
At least with an OpenMPI application, you can achieve task affinity regardless of slurm (https://www.open-mpi.org/faq/?category=tuning#maffinity-defs). However, sharing a node will ALWAYS negatively impact the affinity, as certain hardware resources are shared among different task that are not aware from each other. Therefore, you should at least try to alloc on a per CPU level (not per CORE level, https://slurm.schedmd.com/cpu_management.html#Example8)