Change MCA component build style default to static

[bwbarrett]

This patch series primarily changes the default component build style from dso to static. At the same time, it cleans up some minor nits and clarifies the selection logic when there are conflicting --enable-mca-dso and --enable-mca-static arguments.

[bwbarrett]

@jsquyres You had brought this up a couple months ago. I'm not sure I have strong preferences on whether or not we should take this patch, but here's how we should make the change in component build style.

[jsquyres]

I do remember bringing this up a few months ago (I think after talking with @rhc54 about it...?). I don't remember if there was a reason that I didn't end up filing a PR like this (e.g., a technical or UX reason). ...or were we waiting for The Great 3rd Party Updates to be completed before doing this?

[lanl-ompi]

Can one of the admins verify this patch?

[bwbarrett]

@jsquyres my recollection was that I said I'd look into it and wasn't sure how hard it would be. Turns out, not very :).

[bwbarrett]

@jsquyres your readme changes look good to me. Thanks!

[rhc54]

Do we need/want to port this to PMIx and PRRTE?

[yosefe]

does it mean that if one component depends on external shared library which is not available, then OpenMPI would not be able to ignore just this one component?

[bwbarrett]

does it mean that if one component depends on external shared library which is not available, then OpenMPI would not be able to ignore just this one component?

Yes. Most of our customers don't seem to do this. For those that do, adding --enable-mca-dso= is probably the recommended course of action.

[bwbarrett]

Do we need/want to port this to PMIx and PRRTE?

If we think this is the right default behavior, yes. I think there's an open question about whether we want to do this. There was a question about ability to make the change, and I wanted to show that it wasn't a hard change...

[bosilca]

Both of these concerns are legit (reducing the IO traffic and not breaking loading the OMPI library on missing external dependencies). We can take a middle-ground approach, where we allow components with no external dependencies (BTL sm, tcp, some mpools) to be statically linked in, while everything that has external dependencies remains as dynamic libraries (except when specificaly overwritten with the option described above). We can select the desired behavior either in the configure.m4 or as a file in the component directory (similar to .ignore).

[jsquyres]

Both of these concerns are legit (reducing the IO traffic and not breaking loading the OMPI library on missing external dependencies). We can take a middle-ground approach, where we allow components with no external dependencies (BTL sm, tcp, some mpools) to be statically linked in, while everything that has external dependencies remains as dynamic libraries (except when specificaly overwritten with the option described above). We can select the desired behavior either in the configure.m4 or as a file in the component directory (similar to .ignore).

This is a good idea. I don't know if I want this to always be the behavior -- e.g., for the Cisco distribution of Open MPI, I'm fine if the core Open MPI libraries depend on libfabric. But a generic Linux distribution may make a different choice, for example.

Perhaps --enable-mca-dso can have another value: auto. If the value is auto, then it'll look to the component for guidance on whether it should be compiled as a DSO or static (via an optional m4 directive in the component's configure.ac to indicate which way it should go, and if it's not specified, use a global default). This will allow components to make their own decision if they want to.

[yosefe]

@jsquyres suggestion sounds good to me

[markalle]

TL;DR : I'd propose keeping treematch in its own private mca_topo_treematch.so since it exports a lot of generic symbol names that could cause problems if they appeared in libmpi.so.

I have a testcase proposal
open-mpi/ompi-tests-public#3
that looks at exported symbol names in files like libmpi.so and reports errors if the symbols don't have some recognized prefix that would segregate them from colliding with users's applications.

And I started a PR to change a bunch of symbols to static and/or add ompi_ prefixes, based on how this PR pulls many MCA's into libmpi.so
#8262

But there were objections at least to modifying the treematch MCA. And that MCA in particular is littered with unreasonably generic exported symbol names, like choose, create_work, exchange, get_time, and update_val for example.

So for this PR I'd at least propose treematch be kept in its own mca_topo_treematch.so if fixing its generic symbol names is being declared off limits.

[jjhursey]

We had a discussion today about this PR, and I just wanted to clarify something for myself. We are not eliminating the DSO mechanism, just changing the default for (most or all) of the components in the tree. So I can still build a DSO an drop it into the DSO lib search path and it will try to include it in the framework component loading. Is that correct?

I say this because it is common for us to ship components that, depending on the client's system configuration, that component may not load because of missing libraries. That's ok and we depend on Open MPI's fallback to the next best component in that case. This is another instance where we never want this component to be linked against libmpi.so since the dependent library may not be present and we still want to run.

[rhc54]

@jjhursey that is one of the proposals on the table - but not the only one. 😄

[jsquyres]

@rhc54 and I talked about this on the phone today:

1. We still think that this is a good idea. In current Open MPI master, a typical build on my cluster shows 51 DSOs. Reducing the corresponding number of dlopen()s is probably a Good Idea.
2. Slurping in the components to project libraries without disabling dlopen is not going to completely eliminate filesystem access at process launch and during MPI_INIT, of course, but it will definitely significantly decrease it. This will be mostly felt at mid- to large-scale, but we're assuming that there's a non-zero number of "mid-sized" systems out there that may still have 1GB Ethernet + NFS for their filesystem. This may be a noticeable help for those users.
3. We think that most users will fall into one of the following categories:
   1. Whatever the default is (i.e., they don't care). This is probably the majority of users.
   2. All-static. Large scale users will want this. And per above, this will probably even benefit mid-scale users.
   3. All-DSO. Developers will probably still use this for convenience (e.g., when developing/debugging a single component). Some small number of users may still want this for historical or religious reasons.
   4. Some static and some DSO. This can be useful for users that have a single Open MPI install that spans multiple different clusters, and they want different components loaded for each cluster. We expect this to be a pretty small number of users.
4. Adding another level of defaults (e.g., the auto system I proposed, above) would seem to:
   1. Further complicate the already-complicate m4 code
   2. Potentially violate the Law of Least Surprise for end users
   3. Be difficult to explain to the user.
5. Our opinions:
   1. The defaults of this PR will be good for most users (i.e., static).
   2. This PR gives the ability to go all-DSO for those who want it.
   3. This PR gives the ability to do "mostly static, but a few DSOs" for those who want it.

In short:

• I retract my auto proposal from above. I don't think it's worth it.
• This PR has the potential to improve startup filesystem performance for users.
• Static linking of components also (slightly) improves per-MPI-process memory utilization.
• This PR has CLI options to cover all use cases.
• This PR is good to go. We should merge.

[jjhursey]

Is there a timeout by when we want to try to merge this PR? Just to put a time box on the discussion.

[jsquyres]

Is there a timeout by when we want to try to merge this PR? Just to put a time box on the discussion.

No, not really. The main issue I think is that if we want to do this, we should give our sister projects time to follow our lead (i.e., PMIx / PRRTE).

[jsquyres]

After I posted yesterday, I re-read everything more carefully and was reminded of the treematch symbol pollution issues. That should be addressed; we don't want poorly-named symbols polluting the MPI application namespace. @jjhursey and I talked through this a bit on the phone today. Things I'll look into:

• Symbol pollution for pr8132 #8262 -- @markalle took a shot at this already.
• Can we use a rename.h-style approach to make all the treematch symbols safe? Yes, this has the drawback of editing treematch, but the impact might well be minimal (i.e., just adding #include "rename.h" in each treematch file)
• If we make treematch not be a libtool convenience library, and instead keep it as a separate library that gets installed (e.g., in $pkglibdir) and then have the component link against it as a regular shared library, do we still have the symbol pollution problem? I don't know/remember offhand.

I'll check these things out.

[gpaulsen]

We started a good discussion on the 1/12 call, but ran out of time, so we'll tackle this earlier on 1/19 call for those who are interested.

[gpaulsen]

Failed to get to this on 1/19 due to release branch discussions.

[jsquyres]

The symbol pollution issue is a real one, and we should figure it out. But keep in mind that the symbol pollution problem already exists and isn't because of this PR.

This PR makes the effects of it more pronounced, but the problem already exists.

[markalle]

On the option @jsquyres listed about what if treematch was its own separate library and libmpi was linked against it, I think that still exposes the end user to all the genericly named symbols in libtreeematch.so. We'd be able to say "hey, it's not OUR fault, those symbols aren't in our code" but the user experience would still be bad. If we dlopened libtreematch.so and accessed everything in it via function pointer I believe that would be safe. In the olden days Platform MPI did that for several external libraries like libibverbs. I'd still lean toward the rename.h approach though

[markalle]

@jsquyres also to put on your radar screen, a proposed testcase for generic symbol names being exported: open-mpi/ompi-tests-public#3 It currently accepts a pretty broad list of prefixes.

And even though there might be discussions still about whether to fix this with prefixes or by going all in on DECLSPEC and only exporting a strict list, the testcase wouldn't care about that, it just looks at the results.

[bwbarrett]

Jumping back into this conversation...

We load DSOs into the global scope today (because reasons) and allow customers to build as static libraries, so it seems like we should just fix treematch, rather than screw with defaults.

I think what George is suggesting with allowing components to set a default might make sense, although it's more work than I can commit to doing right now (particularly chasing through setting a proper default for all the components).

To clarify for Josh, this does not change the ability for Open MPI to search for components built out of tree (or similar). This only changes whether components are built as static or dso libraries by default. For at least 10 years, we've split the logic between looking for dsos (the --disable-dlopen option) and how components are built (the --enable-mca-dso and --enable-mca-static options).

[gpaulsen]

@bwbarrett - This looks ready to go. Please PR to v5.0 after merging.