LSF binding is incorrect if multiple app contexts

[jjhursey]

When scheduled under LSF, and with LSF specifying bindings (via the LSB_AFFINITY_HOSTFILE file) if the user runs mpirun with multiple app contexts the binding will be incorrect.

For example, if the mpirun looks like:

mpirun -np 2 myprog1 : -np 2 myprog2 : -np 2 myprog3

And let's assume that we are mapping by socket, and have 4 sockets per node over 2 node (nodeA, nodeB). Open MPI will map:

 nodeA: socket0: myprog1 (rank 0)
 nodeA: socket1: myprog1 (rank 1)
 nodeA: socket0: myprog2 (rank 2)
 nodeA: socket1: myprog2 (rank 3)
 nodeA: socket0: myprog2 (rank 4)
 nodeA: socket1: myprog2 (rank 5)

Instead of what might be expected:

 nodeA: socket0: myprog1 (rank 0)
 nodeA: socket1: myprog1 (rank 1)
 nodeA: socket2: myprog2 (rank 2)
 nodeA: socket3: myprog2 (rank 3)
 nodeB: socket0: myprog2 (rank 4)
 nodeB: socket1: myprog2 (rank 5)

What is happening is that the lsf RAS is associating the same LSB_AFFINITY_HOSTFILE to each of the app_contexts. Then the seq RMAPS is processing each app_context one at a time. When it processes an app_context it is paying attention to where it is in the file (which is why we see the different bindings for 2 of the processes of the first program), but when it switches app_contexts it resets to the beginning since it assumes it is a new file (which is why it goes back over those same bindings).

Note: Affinity was added back in 3f9d9ae (Nov. 2014) where both the RAS and RMAPS were updated.

This was noticed on v1.10.2, but I would assume it would still happen on the v2.x and master branches.

[rhc54]

Ah, my apologies - I do now remember working with someone on this feature. IIRC, the customer was in Europe, but I don't honestly recall if IBM was involved or not.

Anyway, the seq mapper should definitely not be resetting between app_contexts - that is a definite bug! I'll fix that one for you, and ensure it gets PR'd across.

[rhc54]

I just reviewed the seq code, and it does not in fact reset to the beginning for each app_context. It actually continues from the last point. However, it does look at the hostfile and dash host for each app_context and follow those listings.

So it is the LSF RAS that is incorrect. What it should be doing is assigning the LSB_AFFINITY_HOSTFILE as the default hostfile (which would then apply across the app_contexts) instead of assigning it to each app_context as an app-specific hostfile. This is trivial to do - just strdup the LSB_AFFINITY_HOSTFILE name to orte_default_hostfile. It won't affect the allocation at all.

Since I cannot test that component, I'll have to let you make the change.

[jjhursey]

I was thinking that this might be a problem with how the RAS component is setting the hostfile. That seems to be an easy fix. I'll try to get to that today. Thanks for the pointer!

[jjhursey]

For those following this bug: I added two pull requests for the release branches:

• v10.1 branch: ras/lsf: Fix affinity for MPMD jobs running under LSF ompi-release#1092
• v2.x branch: ras/lsf: Fix affinity for MPMD jobs running under LSF ompi-release#1093