Expand communication pages #75
Conversation
|
preview available: https://docs.tds.cscs.ch/75 |
| The instructions found on this page may be inaccurate, but are a good starting point to using OpenMPI on Alps. | ||
|
|
||
| !!! todo | ||
| Deploy experimental uenv. |
There was a problem hiding this comment.
Will do - we don't need to do this before we deploy these docs
| export OMPI_MCA_pml="^ucx" # (4) | ||
| export OMPI_MCA_mtl="ofi" # (5) |
There was a problem hiding this comment.
These are optional in my experience when using just OFI CXI. I mostly used them when experimenting with LINKx and I tried to replicate as close as possible the cmd line from the LINKx paper (i.e. the one from LANL/ORNL about OpenMPI on Cray EX).
There was a problem hiding this comment.
So what you're saying is: if OpenMPI is "correctly" built for Alps (only the necessary providers/backends/etc.) then these two should not be needed? If yes, I agree, we can leave this out.
There was a problem hiding this comment.
I did not need them to get the numbers I was expecting when tinkering with my OFI+CXI+OMPI container.
My guess is that if everything is set up "correctly", OpenMPI evaluates that OFI is the best layer to use and things go well.
But on second thought, having these vars around would be helpful to "anchor" OMPI into the behavior we want. Otherwise we have little clue if/when something changes and e.g. OMPI chooses different components/backends for whatever reason.
All of this obviously assumes that we recognize what we get with these vars as the "accepted and expected" behavior/performance.
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
| See [the container engine documentation][ref-ce-aws-ofi-hook] for information on using NCCL in containers. | ||
| The [NCCL][ref-communication-nccl] contains general information on configuring NCCL. | ||
| This information is especially important when using uenvs, as the environment variables are not set automatically. |
There was a problem hiding this comment.
NB @bcumming, I'm linking from the GB docs to the updated NCCL docs now.
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
Co-authored-by: Rocco Meli <r.meli@bluemail.ch>
Co-authored-by: Rocco Meli <r.meli@bluemail.ch>
|
preview available: https://docs.tds.cscs.ch/75 |
1 similar comment
|
preview available: https://docs.tds.cscs.ch/75 |
Co-authored-by: Rocco Meli <r.meli@bluemail.ch>
|
preview available: https://docs.tds.cscs.ch/75 |
Co-authored-by: Rocco Meli <r.meli@bluemail.ch>
|
preview available: https://docs.tds.cscs.ch/75 |
1 similar comment
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
| export FI_CXI_DISABLE_HOST_REGISTER=1 | ||
| export FI_CXI_RX_MATCH_MODE=software | ||
| export FI_MR_CACHE_MONITOR=userfaultfd | ||
| export MPICH_GPU_SUPPORT_ENABLED=0 # (4)! |
There was a problem hiding this comment.
| export MPICH_GPU_SUPPORT_ENABLED=0 # (4)! | |
| export MPICH_GPU_SUPPORT_ENABLED=0 # (4)! | |
| export NCCL_SOCKET_IFNAME="hsn" | |
| export NCCL_CROSS_NIC="1" | |
| export FI_CXI_COMPAT="0" | |
| export MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD=100000000 |
@boeschf, @teojgo, @fawzi (or anyone else that knows): any comments on NCCL_SOCKET_IFNAME, NCCL_CROSS_NIC, FI_CXI_COMPAT, and MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD? They are set by enroot in addition to the environment variables that we already have documented here. Can we safely recommend them to users?
There was a problem hiding this comment.
MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLD only affects some MPICH-based MPIs; MPICH and MVAPICH respond to it, but beyond that I'm not sure.
There was a problem hiding this comment.
For reference, the default vars set by the AWS NCCL plugin hook (which reasonably apply only when NCCL is used) are defined here: https://git.cscs.ch/alps-platforms/vservices/vs-enroot/-/blob/main/enroot-variables.tf?ref_type=heads#L314
Individual vclusters might override these values, of course.
There was a problem hiding this comment.
MPIR_CVAR_CH4_OFI_MULTI_NIC_STRIPING_THRESHOLDonly affects some MPICH-based MPIs; MPICH and MVAPICH respond to it, but beyond that I'm not sure.
Ah, of course 🤦 makese sense. So definitely not for the NCCL page then. Might still be useful to add to the Cray MPICH page, but I'd like to first understand how universally useful it is.
For reference, the default vars set by the AWS NCCL plugin hook (which reasonably apply only when NCCL is used) are defined here: https://git.cscs.ch/alps-platforms/vservices/vs-enroot/-/blob/main/enroot-variables.tf?ref_type=heads#L314
Indeed, thanks for the link. That's what I was using as a reference to understand if we need to list all of them on the NCCL page here in the docs. Based on docs FI_CXI_COMPAT and NCCL_SOCKET_IFNAME seem to be more for safety, and should be ok to set. NCCL_CROSS_NIC seems a bit more unclear. It definitely has an effect on performance, but it's not clear if it's always good or bad.
|
preview available: https://docs.tds.cscs.ch/75 |
|
preview available: https://docs.tds.cscs.ch/75 |
bcumming
left a comment
bcumming
left a comment
There was a problem hiding this comment.
This is a good start on these docs.
It covers a range of topics - so in the interest of avoiding multiple bike sheds, let's merge then work on the separate topics individually.
5 participants
Adds a bit more text to the libfabric, OpenMPI, NCCL, and RCCL pages. Still far from complete, but another step forward hopefully.