Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change MCA component build style default to static #8132

Merged
merged 3 commits into from
Mar 25, 2021

Conversation

bwbarrett
Copy link
Member

This patch series primarily changes the default component build style from dso to static. At the same time, it cleans up some minor nits and clarifies the selection logic when there are conflicting --enable-mca-dso and --enable-mca-static arguments.

@bwbarrett
Copy link
Member Author

@jsquyres You had brought this up a couple months ago. I'm not sure I have strong preferences on whether or not we should take this patch, but here's how we should make the change in component build style.

Copy link
Member

@jsquyres jsquyres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do remember bringing this up a few months ago (I think after talking with @rhc54 about it...?). I don't remember if there was a reason that I didn't end up filing a PR like this (e.g., a technical or UX reason). ...or were we waiting for The Great 3rd Party Updates to be completed before doing this?

config/opal_mca.m4 Outdated Show resolved Hide resolved
@lanl-ompi
Copy link
Contributor

Can one of the admins verify this patch?

@bwbarrett
Copy link
Member Author

@jsquyres my recollection was that I said I'd look into it and wasn't sure how hard it would be. Turns out, not very :).

README Outdated Show resolved Hide resolved
@bwbarrett
Copy link
Member Author

@jsquyres your readme changes look good to me. Thanks!

@rhc54
Copy link
Contributor

rhc54 commented Oct 28, 2020

Do we need/want to port this to PMIx and PRRTE?

@yosefe
Copy link
Contributor

yosefe commented Oct 28, 2020

does it mean that if one component depends on external shared library which is not available, then OpenMPI would not be able to ignore just this one component?

@bwbarrett
Copy link
Member Author

does it mean that if one component depends on external shared library which is not available, then OpenMPI would not be able to ignore just this one component?

Yes. Most of our customers don't seem to do this. For those that do, adding --enable-mca-dso= is probably the recommended course of action.

@bwbarrett
Copy link
Member Author

Do we need/want to port this to PMIx and PRRTE?

If we think this is the right default behavior, yes. I think there's an open question about whether we want to do this. There was a question about ability to make the change, and I wanted to show that it wasn't a hard change...

@bosilca
Copy link
Member

bosilca commented Oct 28, 2020

Both of these concerns are legit (reducing the IO traffic and not breaking loading the OMPI library on missing external dependencies). We can take a middle-ground approach, where we allow components with no external dependencies (BTL sm, tcp, some mpools) to be statically linked in, while everything that has external dependencies remains as dynamic libraries (except when specificaly overwritten with the option described above). We can select the desired behavior either in the configure.m4 or as a file in the component directory (similar to .ignore).

@jsquyres
Copy link
Member

Both of these concerns are legit (reducing the IO traffic and not breaking loading the OMPI library on missing external dependencies). We can take a middle-ground approach, where we allow components with no external dependencies (BTL sm, tcp, some mpools) to be statically linked in, while everything that has external dependencies remains as dynamic libraries (except when specificaly overwritten with the option described above). We can select the desired behavior either in the configure.m4 or as a file in the component directory (similar to .ignore).

This is a good idea. I don't know if I want this to always be the behavior -- e.g., for the Cisco distribution of Open MPI, I'm fine if the core Open MPI libraries depend on libfabric. But a generic Linux distribution may make a different choice, for example.

Perhaps --enable-mca-dso can have another value: auto. If the value is auto, then it'll look to the component for guidance on whether it should be compiled as a DSO or static (via an optional m4 directive in the component's configure.ac to indicate which way it should go, and if it's not specified, use a global default). This will allow components to make their own decision if they want to.

@jsquyres jsquyres marked this pull request as draft October 29, 2020 11:33
@yosefe
Copy link
Contributor

yosefe commented Oct 29, 2020

@jsquyres suggestion sounds good to me

@markalle
Copy link
Contributor

markalle commented Dec 2, 2020

TL;DR : I'd propose keeping treematch in its own private mca_topo_treematch.so since it exports a lot of generic symbol names that could cause problems if they appeared in libmpi.so.

I have a testcase proposal
open-mpi/ompi-tests-public#3
that looks at exported symbol names in files like libmpi.so and reports errors if the symbols don't have some recognized prefix that would segregate them from colliding with users's applications.

And I started a PR to change a bunch of symbols to static and/or add ompi_ prefixes, based on how this PR pulls many MCA's into libmpi.so
#8262

But there were objections at least to modifying the treematch MCA. And that MCA in particular is littered with unreasonably generic exported symbol names, like choose, create_work, exchange, get_time, and update_val for example.

So for this PR I'd at least propose treematch be kept in its own mca_topo_treematch.so if fixing its generic symbol names is being declared off limits.

@jjhursey
Copy link
Member

jjhursey commented Dec 3, 2020

We had a discussion today about this PR, and I just wanted to clarify something for myself. We are not eliminating the DSO mechanism, just changing the default for (most or all) of the components in the tree. So I can still build a DSO an drop it into the DSO lib search path and it will try to include it in the framework component loading. Is that correct?

I say this because it is common for us to ship components that, depending on the client's system configuration, that component may not load because of missing libraries. That's ok and we depend on Open MPI's fallback to the next best component in that case. This is another instance where we never want this component to be linked against libmpi.so since the dependent library may not be present and we still want to run.

@rhc54
Copy link
Contributor

rhc54 commented Dec 3, 2020

@jjhursey that is one of the proposals on the table - but not the only one. 😄

@jsquyres
Copy link
Member

jsquyres commented Jan 7, 2021

@rhc54 and I talked about this on the phone today:

  1. We still think that this is a good idea. In current Open MPI master, a typical build on my cluster shows 51 DSOs. Reducing the corresponding number of dlopen()s is probably a Good Idea.
  2. Slurping in the components to project libraries without disabling dlopen is not going to completely eliminate filesystem access at process launch and during MPI_INIT, of course, but it will definitely significantly decrease it. This will be mostly felt at mid- to large-scale, but we're assuming that there's a non-zero number of "mid-sized" systems out there that may still have 1GB Ethernet + NFS for their filesystem. This may be a noticeable help for those users.
  3. We think that most users will fall into one of the following categories:
    1. Whatever the default is (i.e., they don't care). This is probably the majority of users.
    2. All-static. Large scale users will want this. And per above, this will probably even benefit mid-scale users.
    3. All-DSO. Developers will probably still use this for convenience (e.g., when developing/debugging a single component). Some small number of users may still want this for historical or religious reasons.
    4. Some static and some DSO. This can be useful for users that have a single Open MPI install that spans multiple different clusters, and they want different components loaded for each cluster. We expect this to be a pretty small number of users.
  4. Adding another level of defaults (e.g., the auto system I proposed, above) would seem to:
    1. Further complicate the already-complicate m4 code
    2. Potentially violate the Law of Least Surprise for end users
    3. Be difficult to explain to the user.
  5. Our opinions:
    1. The defaults of this PR will be good for most users (i.e., static).
    2. This PR gives the ability to go all-DSO for those who want it.
    3. This PR gives the ability to do "mostly static, but a few DSOs" for those who want it.

In short:

  • I retract my auto proposal from above. I don't think it's worth it.
  • This PR has the potential to improve startup filesystem performance for users.
  • Static linking of components also (slightly) improves per-MPI-process memory utilization.
  • This PR has CLI options to cover all use cases.
  • This PR is good to go. We should merge.

@jsquyres jsquyres marked this pull request as ready for review January 7, 2021 20:15
@jjhursey
Copy link
Member

jjhursey commented Jan 7, 2021

Is there a timeout by when we want to try to merge this PR? Just to put a time box on the discussion.

@jsquyres
Copy link
Member

jsquyres commented Jan 7, 2021

Is there a timeout by when we want to try to merge this PR? Just to put a time box on the discussion.

No, not really. The main issue I think is that if we want to do this, we should give our sister projects time to follow our lead (i.e., PMIx / PRRTE).

@jsquyres
Copy link
Member

jsquyres commented Jan 8, 2021

After I posted yesterday, I re-read everything more carefully and was reminded of the treematch symbol pollution issues. That should be addressed; we don't want poorly-named symbols polluting the MPI application namespace. @jjhursey and I talked through this a bit on the phone today. Things I'll look into:

  • Symbol pollution for pr8132 #8262 -- @markalle took a shot at this already.
  • Can we use a rename.h-style approach to make all the treematch symbols safe? Yes, this has the drawback of editing treematch, but the impact might well be minimal (i.e., just adding #include "rename.h" in each treematch file)
  • If we make treematch not be a libtool convenience library, and instead keep it as a separate library that gets installed (e.g., in $pkglibdir) and then have the component link against it as a regular shared library, do we still have the symbol pollution problem? I don't know/remember offhand.

I'll check these things out.

@gpaulsen
Copy link
Member

We started a good discussion on the 1/12 call, but ran out of time, so we'll tackle this earlier on 1/19 call for those who are interested.

@gpaulsen
Copy link
Member

Failed to get to this on 1/19 due to release branch discussions.

@jsquyres
Copy link
Member

The symbol pollution issue is a real one, and we should figure it out. But keep in mind that the symbol pollution problem already exists and isn't because of this PR.

This PR makes the effects of it more pronounced, but the problem already exists.

@gpaulsen gpaulsen added this to the v5.0.0 milestone Jan 21, 2021
@markalle
Copy link
Contributor

On the option @jsquyres listed about what if treematch was its own separate library and libmpi was linked against it, I think that still exposes the end user to all the genericly named symbols in libtreeematch.so. We'd be able to say "hey, it's not OUR fault, those symbols aren't in our code" but the user experience would still be bad. If we dlopened libtreematch.so and accessed everything in it via function pointer I believe that would be safe. In the olden days Platform MPI did that for several external libraries like libibverbs. I'd still lean toward the rename.h approach though

@markalle
Copy link
Contributor

@jsquyres also to put on your radar screen, a proposed testcase for generic symbol names being exported: open-mpi/ompi-tests-public#3 It currently accepts a pretty broad list of prefixes.

And even though there might be discussions still about whether to fix this with prefixes or by going all in on DECLSPEC and only exporting a strict list, the testcase wouldn't care about that, it just looks at the results.

@bwbarrett
Copy link
Member Author

Jumping back into this conversation...

We load DSOs into the global scope today (because reasons) and allow customers to build as static libraries, so it seems like we should just fix treematch, rather than screw with defaults.

I think what George is suggesting with allowing components to set a default might make sense, although it's more work than I can commit to doing right now (particularly chasing through setting a proper default for all the components).

To clarify for Josh, this does not change the ability for Open MPI to search for components built out of tree (or similar). This only changes whether components are built as static or dso libraries by default. For at least 10 years, we've split the logic between looking for dsos (the --disable-dlopen option) and how components are built (the --enable-mca-dso and --enable-mca-static options).

@bwbarrett bwbarrett closed this Mar 16, 2021
@bwbarrett bwbarrett deleted the feature/3rdparty-packaging branch March 16, 2021 17:23
opal_mca.m4 was setting enable_dlopen after it was processed, and
the dlopen argument checking was overriding configure arguments
that belong to opal_mca.  Clean up the late change in enable_dlopen
and handle the --disable-dlopen case in opal_mca instead of
overriding user options.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
Remove an environment variable that probably did something at one
point, but is dead code now.  Git history is hard to follow due
to some reverts in the SVN days.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
Default to building MCA components into the library, rather than
as dso objects, since most users are not taking advantage of
per-library packaging, and this results in less stat/open/etc. calls
during application startup.

Clean up the static/dso selection logic for componets at the same time.
Before, it was relatively hard to describe what would happen if
there was a mix of --enable-mca-dso and --enable-mca-static arguments.
Now, the code will first look for a component-specific argument, then
a framework-level argument, then a global argument.  If there is a tie,
static is preferred.

Update the NEWS and README.md to document the changes.  Jeff Squyres
wrote the original update to the README file, which was adopted to
the new README.md format.

Signed-off-by: Brian Barrett <bbarrett@amazon.com>
@bwbarrett bwbarrett restored the feature/3rdparty-packaging branch March 16, 2021 17:36
@bwbarrett bwbarrett reopened this Mar 16, 2021
@gpaulsen
Copy link
Member

@bwbarrett - This looks ready to go. Please PR to v5.0 after merging.

@bwbarrett bwbarrett merged commit 856a2b7 into open-mpi:master Mar 25, 2021
@bwbarrett bwbarrett deleted the feature/3rdparty-packaging branch April 1, 2021 18:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

9 participants