Incorrect block mapping for cyclic MPI job distribution across nodes #166
Comments
vasdommes
added a commit
that referenced
this issue
Jan 3, 2024
…ss nodes Use ranks from node communicator instead of global ranks (which can be assigned to nodes in different ways).
vasdommes
added a commit
that referenced
this issue
Jan 3, 2024
…ss nodes Use ranks from node communicator instead of global ranks (which can be assigned to nodes in different ways).
vasdommes
added a commit
that referenced
this issue
Jan 5, 2024
Fix #166 Incorrect block mapping for cyclic MPI job distribution across nodes
bharathr98
pushed a commit
to bharathr98/sdpb
that referenced
this issue
Mar 1, 2024
…on across nodes Use ranks from node communicator instead of global ranks (which can be assigned to nodes in different ways).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
When assigning blocks, SDPB assumes that first
procsPerNodeMPI ranks belong to the first node.In SLURM, this corresponds to
--distribution=blockoption forsrun, see https://slurm.schedmd.com/srun.html#OPT_distributionFor other distributions, this is not true. For example, if
--distribution=cyclicoption is set instead, then first rank goes to the first node, second rank - to the second node etc.In that case, if some block is shared among rank 0 and rank 1, its data is spread among different nodes.
For SDPB 2.6.1, this could lead to slow
DistMatrixoperations (e.g. cholesky decomposition) due to slow communication between nodes.For future versions using shared memory window (e.g. after merging #142), this will lead to errors and/or failures.
The text was updated successfully, but these errors were encountered: