Issue in MemoryUsageReporter

[friedmud]

Rationale

Shared memory group hardware_id assignment might not be right in the case where ranks aren't perfectly monotonic across the nodes.

Description

MemoryUsageReporter::sharedMemoryRanksBySplitCommunicator() is close - but not quite right. The part where it loops through the "contiguous" ranks and assigns them... assumes that ranks on one node are contiguous! That might not actually be true.

For intance... it's totally possible to do this:

mpiexec -hosts host0,host1,host1,host0 ...

In this case sharedMemoryRanksBySplitCommunicator() would report 3 hardware IDs.

Now - you might think that's completely dumb... but one common thing that happens with mpiexec is "striping"... where processes are spread across the nodes by adding one to each node in turn. That can happen if you do something like:

mpiexec -hosts host0,host1,host2 -n 6 ...

In that case you'll get this node:rank mapping:

0: 0,3
1: 1,4
2: 2,5

Which would also throw off sharedMemoryRanksBySplitCommunicator().

The fix here is really just to keep track of the "world_ranks" and what ID they've been assigned and check it each time it changes to see if you've already seen this rank before.

Actually - to fix this I might make a new utility that creates node->rank and rank->node mappings (I need both for my current work - which is why I was looking at what was done here). Then I'll make MemoryUsageReporter use the new utility.

Impact

Correct hardware ID reporting.

Fix a small bug that probably doesn't effect anyone.

[dschwen]

Good catch!