TL/UCP: ranks reordering

[Sergei-Lebedev]

What

Use topology information in TL UCP to reorder ranks so that ranks first grouped by node, by socket within node, by numa within socket.

Why ?

Permutation is used in TL UCP allgather to form better ring algorithm.
Performance data collected on N2 PPN48 (AMD EPYC 7443: 24 cores, 2 sockets, 8 NUMAs)

[bureddy]

is it host list order preserved when doing this ordering?

[Sergei-Lebedev]

it sorts hosts by hashes, for instance
original mapping:
rank0 - hostA (hash: 0xB) socket 0
rank1 - hostA (hash: 0xB) socket 1
rank2 - hostA (hash: 0xB) socket 0
rank3 - hostA (hash: 0xB) socket 1
rank4 - hostB (hash: 0xA) socket 0
rank5 - hostB (hash: 0xA) socket 1
rank6 - hostB (hash: 0xA) socket 0
rank7 - hostB (hash: 0xA) socket 1
reordering:
rank4, rank6, rank5, rank7, rank0, rank2, rank1, rank3

[vspetrov]

do we need this flag? the original idea was that subset is set to EP_MAP_FULL by default. So if collective supports it, then it always uses "subset" which evaluates to no-op for FULL. Otherwise works as expected.

[Sergei-Lebedev]

it's used to identify if subset is already set and shouldn't be changed. For example, in service collective.

[vspetrov]

you should use ucc_ep_map_from_array to try to pack the ep map (detect strided/flat). Alloc int *rank, fill in line 430, and then ucc_ep_map_from_array.

[Sergei-Lebedev]

yes, it's already used in ucc_sbgp_create, but it can't detect map by numa, socket or node because they are not really strided

[vspetrov]

we could maybe try to remove that conditionals for services vs regular. IF we introduce tl_ucp_team->reorder_map explciitly rathar than updating subset. Then:

allgather_ring_init() {
     if (!team_is_single_node(tl_ucp_team) && cfg->allgather_use_host_ordering) {
             if (task->subset == FULL) {
                   sbgp = get_sbgp(HOST_ORDERED);
                   task->reorder_map = sbgp.map;
             } else {
                   task->reorder_map = FULL; // this is true for service 
             }
     }

then in the collective you would use double mapping team->subset->reorder_map ? Now that i think of it, we maybe even can udpate subset as ucc_ep_map_create_nested(subset, reorder_map) and it will work? (need to check)

[Sergei-Lebedev]

not sure i understand, semantic is diffrrent for service allgather and regular allgather. Service allgather with subset looks to me like a lightweight allgather with new subteam and so new topo based permutation required in that case.

[vspetrov]

yeah.. i was thinking how to avoid that difference of flows between service and regular. At first i thought we can unify them, but looks like we can't. we need separate maps for "peer" and "block" computations for the 2 cases. so basically i meant to be explcit: 1 thing is "subset", it running an algorithm over only a handful of procs out of the team; another thin is "reordering".

Suppose we would introduce "reorder_map" default to FULL, optionally set to get_sbgp(HOST_ORDERED).map. Then in your allgather implementation you would have:

    sendto   = ucc_ep_map_eval(task->subset.map, (trank + 1) % tsize);
    recvfrom = ucc_ep_map_eval(task->subset.map, (trank - 1 + tsize) % tsize);
    sendto   = ucc_ep_map_eval(task->reorder.map, sendto);
    recvfrom = ucc_ep_map_eval(task->reorder.map, recvfrom);

   ...
   get_recv_block {
     ...
   ucc_ep_map_eval(task->reorder_map, (trank - step - 1 + tsize) % tsize);
   }

So, this way you don't have fn pointers for send/get block. the behavior is achieved by setting reorder_map = FULL for service. Then it would even could be possible to support reordering for subset (though i don't see a use case for that now. maybe if we would like to support arbitrary size active sets we could use re-ordering for them).

Anyways, this was my logic. And when i wrote this i quickly thought that maybe this even allows to build nested map (my comment above) - but that was incorrect.

[Sergei-Lebedev]

code snippet above is not correct, subset.map is actually reverse permutation that map from new numbering to old numbering, subset.myrank is rank number in new numbering. In ring algorithm we first need to find neighbors in new numbering and then translate it back to old numbering to know EPs. However, if we want to combine multiple permutations only reverse permutation is not enough, we also need forward permutation that translates from old numbering to new numbering.
Since we are not going to use subset and reordering simultaneously anyway, maybe add union to tl_ucp task structure:

union {
  ucc_subset_t subset;
  ucc_subset_t reordering;
} 

If algorithm supports both subset and reordering it will use subset, otherwise it will use reordering field. In those algorithms that support both we will need somehow to check if rank permutation is enough, or we also need block permutation. This can be done by providing function pointers to blocks call functions or adding flag that says to permute ranks only and don't touch blocks.

[manjugv]

@vspetrov are you ok with the changes?

[manjugv]

ping @vspetrov

[vspetrov]

2023

[vspetrov]

2023

[vspetrov]

2023

[vspetrov]

2023

[Sergei-Lebedev]

bot:retest