-
Notifications
You must be signed in to change notification settings - Fork 41
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Can we allow for site-specific tuning of OpenMPI? #456
Comments
We need something similar but we also need to allow for other fabrics (like EFA) |
This is another reason to let sites provide a script that is sourced along with initializing the EESSI environment, although in this case we should probably try and auto-detect which environment variables we should set? It's a bit silly we have to do that though, why isn't OpenMPI doing that? |
We could file bugs to Open MPI for some subcases I presume. When Open MPI is compiled with loadable runtime plugins there's also some benefit in memory and start-up time, not loading the .so with the associated libraries it links to (libfabric.so etc), which can't be fixed otherwise. |
I've run into this with the OSU tests with CUDA support as well:
There is more than one thing going on here by the way. Setting:
I get fewer errors, but the code still doens't run:
We found the same issue in our local module stack, and found that we have to:
Before this thing runs as it should:
@satishskamath is a bit deeper into the details, but the gist of that UCX issue is that it tries to use the TCP interface on the public IP address of our nodes. This fails, because the firewall closes this off. Similar to the OpenMPI case above, I guess this could be considered a bug in initialization and priority mechanisms: maybe it shouldn't have picked TCP to begin with, but even if it does, one would hope that if that initialization fails it would fall back to another mechanism. I'll ask Satish to report this upstream, but I guess the bottom line from an EESSI point of view is: we will have these system-specific settings that a hosting site might want to change, whether they are unresolved bugs, tuning paramters, or otherwise. I'm wondering what the most convenient way is. I think we should have a good discussion on this, and then document it. From Kenneths comment:
A script that gets sourced would allow some global config, but some of this may be specific to which version of e.g. UCX or OpenMPI gets loaded. I'm no expert in LMOD hooks, but I'm wondering if these shouldn't be the solution here. From what I've seen when implementing our GPU support, I think one can control how specific they are (from "set X for all modules named Y" to "set X for modules named Y if their version is Z"). Aside from resolving bugs, the same mechanism could even facilitate site-specific OpenMPI tuning. What do you think? Would LMOD hooks be a good practice? If so, we can use the current use case to as an example on how to write such a site-specific LMOD hook, how we should make sure it gets picked up, and document that procedure. |
Just checked our hook. It is defined in the
That's where we define the hooks currently:
I think there can only be one LMOD_RC file, but we could source something from |
Good find from @ocaisa : https://lmod.readthedocs.io/en/latest/145_properties.html?highlight=lmod_rc#the-properties-file-lmodrc-lua talks about the search order of Lmod RC files. More importantly, it says:
That means that configuration can be set at multiple levels, and will be merged. The only thing we should consider is what we want to happen in conflicting situations. The search order is:
meaning that if the Also important: we should probably prefix anything we do with an |
To be kept in mind: hooks should be defined in |
Just to log somewhere: I'm running into the same issue as #456 (comment) on the Karolina HPC cluster as well. The job:
(A test from this pr to the EESSI test suite) And the output file is:
We should contact the Karolina staff (and probably also the Vega staff) to see if they can provision some (permanent) storage for us to host the |
…fix it through an LMOD hook in host_injections
I ran into this issue that a simple mpi4py code would not run on a Magic Castle deployment with EESSI (though it works on the same system with the pilot):
It turns out this issue was already "solved" for an EasyBuild use case, which also resolved things for my case.
It raises the issue though that OpenMPI may need to be configured to work correctly on the host site (and indeed this was also raised in #1 ). @bartoldeman explained how they account for this in Compute Canada:
The text was updated successfully, but these errors were encountered: