Implement --enable-static-runtime configure option. #818
Comments
|
@gpaulsen Your Github wiki-foo is weak. :-) I updated the description to be proper wiki notation. |
|
😳 I shall aspire to wiki-it-up ❗ |
|
fwiw, except having a statically linked orted (or orted that only links with system libs) i do not see how this can be achieved ... |
|
That is correct - Brian Barrett had done this a few years back. The key is that you build all of OMPI as static. Today, that still won't build the orted as fully static, so you have to add some magic dust to get that to include all the dependent libs. I don't remember what he had to do - something in the orted Makefile.am plus some configure stuff. You could send him a note and ask - he might remember. |
|
Why must all of OMPI be built static? Why not just link / archive (ar) each of the OMPI pieces against the static compiler runtime libraries? seems like you could still build a liborte.so, but just archive in the static C runtime. |
|
You could try that - I was just relaying what Brian had done. The problem he had was that he built OMPI static, but the orted was not fully static and was still trying to load dependent libs. |
|
@gpaulsen is that even possible ? if i read you correctly, it means each .so includes the static compiler runtime lib. that would make all the .so very big, plus all symbols would be defined multiple time. |
|
I'd think it'd be possible if you took all of libX's .os and ar'ed them with the compiler runtime .a to create another .a (built PIC of course), and then created a .so from that. I'd think the ar would resolve any of the undefines, but not pickup any symbols that were not needed. |
|
Remember: the goal here is not to make a 100% static orted, but rather just eliminate the need for setting LD_LIBRARY_PATH on remote nodes to find the compiler run-time libraries. The mechanism for doing this -- with the Intel compiler, at least -- just happens to be |
|
I'm not sure that is completely accurate. We want to eliminate dependencies on any non-standard dynamic libraries. So if you build OMPI against an external hwloc, libevent, or libpmix, then we will also want to pick those up since they might not be installed in the remote LD_LIBRARY_PATH. Given all the potential combinations, a static orted might be the easiest solution. |
|
FWIW, I stated that the goal was to link in the compiler runtime because we know we can do that. It's easy, low-hanging fruit, and is >90% of the problems we've heard over the years. It's difficult to make a 100% static
|
|
I made some tests today, and surprisingly, linking orted with --static is enough, assuming ompi was configure'd with --enable-static. it seems --static-intel is not even necessary since the generated c code does not seem to require the runtime (!) |
|
I'd guess that in the Intel compiler suite, if they see I agree that it might be problematic to ensure that if |
An Huang on our team came to the conclusion that this is not a low hanging fruit...Intel has a link https://software.intel.com/en-us/articles/intel-c-compiler-for-linux-statically-linking-intel-provided-libraries-into-your-application says: By default, some Intel provided libraries are linked dynamically into your application, such as libcxaguard.so. This may not be desired, if you plan to run your application on a system that does not have the Intel C++ Compiler installed on it or you do not want to redistribute the Intel provided libraries with your application. But actually, -static-intel cannot work as expected. I added -static to CFLAGS and CXXFLAGS into Makefile under orte and opal, rebuilt them successfully, but they fail at runtime: building hello.c still has problems:[anzhang@mpi04 examples]$ make hello_c [anzhang@mpi04 examples]$ mpirun -np 4 ./hello_c Then I tried to build the libmpi.so with LDFLAGS=-static -static-intel, failed with the following problem: Building with latest intel compiler still has issues
|
|
@gpaulsen I don't think your tests were quite right: you need to link Here's what I did: $ ./configure --prefix=/home/jsquyres/bogus CC=icc CXX=icpc FC=ifort ...
$ make -j 32
# ...
$ cd orte/tools/orted
$ rm orted
$ make install LDFLAGS=-static-intel
# ...
$ cd ../orterun
$ rm orterun
$ make install LDFLAGS=-static-intel
# ...With these steps, I get a fully functional Open MPI installation (including All that being said, it looks like the Intel compiler does not link in its runtime statically (even though I verified manually above via $ cd installation_tree/bin
libimf.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libimf.so (0x00002aaaab63b000)
libsvml.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libsvml.so (0x00002aaaabaf7000)
libirng.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libirng.so (0x00002aaaac9ca000)
libintlc.so.5 => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libintlc.so.5 (0x00002aaaacbd1000)I can even see this when compiling MPI "hello world": # Plain build
$ mpicc hello_c.c -o hello
$ ldd hello | grep intel
libimf.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libimf.so (0x00002aaaabaef000)
libsvml.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libsvml.so (0x00002aaaabfaa000)
libirng.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libirng.so (0x00002aaaace7d000)
libintlc.so.5 => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libintlc.so.5 (0x00002aaaad085000)
# Build with -static-intel
$ mpicc hello_c.c -o hello -static-intel
icc: warning #10237: -lcilkrts linked in dynamically, static library not available
$ ldd hello | grep intel
libimf.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libimf.so (0x00002aaaabaef000)
libsvml.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libsvml.so (0x00002aaaabfaa000)
libirng.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libirng.so (0x00002aaaace7d000)
libintlc.so.5 => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libintlc.so.5 (0x00002aaaad085000)You can see that building "hello world" with and without Perhaps this is an Intel compiler bug...? |
|
@jsquyres surely you jest... :-) |
|
@hppritcha is this something you could ask your friends there at the COE? |
|
Hello, this is An Huang :) $ make install LDFLAGS=-static-intel $ ldd /pmpi2/anzhang/src/ompi.newest.icc/run/common/bin/orted |
|
as discussed in #829 can you try to configure with |
|
Ah, you're correct!, @ggouaillardet , thanks! |
|
@ggouaillardet They're testing with setting LDFLAGS at build time because the suggestion was to just set those flags for just the That being said, $ mpicc hello_c.c -o hello -Wc,-static-intel
icc: command line warning #10006: ignoring unknown option '-Wc,-static-intel'I have a fairly recent version of the Intel compiler: $ icc -V
Intel(R) C Intel(R) 64 Compiler XE for applications running on Intel(R) 64, Version 15.0.3.187 Build 20150407
Copyright (C) 1985-2015 Intel Corporation. All rights reserved. |
|
got it, thanks |
|
@ggouaillardet See my comment, above -- even when I pass Additionally, when I passed $ cd orte/tools/orted
$ rm orted
$ make V=1 LDFLAGS=-static-intel
/bin/sh ../../../libtool --tag=CC --mode=link icc -std=gnu99 -g -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types -pthread -static-intel -o orted orted.o ../../../orte/libopen-rte.la ../../../opal/libopen-pal.la -lrt -lutil -lrt -lutil
libtool: link: icc -std=gnu99 -g -finline-functions -fno-strict-aliasing -restrict -Qoption,cpp,--extended_float_types -pthread -static-intel -o .libs/orted orted.o ../../../orte/.libs/libopen-rte.so /home/jsquyres/git/ompi/opal/.libs/libopen-pal.so ../../../opal/.libs/libopen-pal.so -lm -lnuma -lpciaccess -ldl -lrt -lutil -pthread -Wl,-rpath -Wl,/home/jsquyres/bogus/lib
icc: warning #10237: -lcilkrts linked in dynamically, static library not available
$ ldd .libs/orted | grep intel
libimf.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libimf.so (0x00002aaaab63b000)
libsvml.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libsvml.so (0x00002aaaabaf7000)
libirng.so => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libirng.so (0x00002aaaac9ca000)
libintlc.so.5 => /opt/intel/composer_xe_2015.3.187/compiler/lib/intel64/libintlc.so.5 (0x00002aaaacbd1000) |
|
iirc, the root cause is libopen-rte.so and libopen-pal.so do require the intel runtime |
|
@ggouaillardet Good point: if I Soo.... I don't know where to go from here. |
|
@jsquyres i do not observe this behavior ... here is some basic info about my system icc version make log note the warning about and the ldd output and here is my super verbose output did you configure with i will give an other try with a more recent compiler |
|
i had no issue with icc 15.0.3 and 16.0.0 |
|
I used the newest icc, there is no problem for statical linking, all orte* binaries are good, since orted depends on libopen-pal.so and libopen-rte.so, these two files needs to be statically linked with intel libs. $ icc -v $./configure CC=icc CXX=icpc FC=ifort LDFLAGS=-Wc,-static-intel ]$ ldd orte* $ ldd libopen-rte.so.7 libopen-pal.so.6 |
|
One strange thing is the Intel option "-static-intel", according to https://software.intel.com/zh-cn/node/523256, it is a linker option, so we should specify it for make with "-Wl" if we want to do a dynamic building for orted, but actually it fails, we have to use "-Wc" [orted]$ make LDFLAGS=-Wl,-static-intel Only -Wc,-static-intel works fine: [orted]$ make LDFLAGS=-Wc,-static-intel Anyway, let's use LDFLAGS=-Wc,-static-intel. |
|
my 0.02US$... |
|
or that could be an icc bug/feature I will double check that tomorrow, meanwhile feel free to ask icc folks what they think of that |
|
my guess was right does invoke whereas does invoke which ends up invoking as i stated earlier, i am not sure whether this is a bug or a feature, and if this is a bug, i cannot tell if it comes from libtool (e.g. do not pass |
|
@ggouaillardet Per
I just tested again on a clean tree. I don't know what happened with my prior test -- I must have had something stale left over. Specifically: when I tested on a clean tree this morning, I got the expected result:
So. Where does this leave us? Since the initial suggested approach of "just link
|
|
In general I'm for simplifying the configure time options, in favor of "doing as much at runtime as possible". Of course that's not an option in this case. We might need to implement this for other Linux Comiler runtime environments as well. It's safe to assume (is it?) that the GNU compiler runtimes would be installed, but other compilers such as the IBM XL C/C++ runtime for Linux on Power (and others that I'm unfamiliar with) should be supported as well. This feels like one of those options that might be a nightmare maintaining over time. Are many interested in this feature? I worry that it might expose other issues in loading both the compiler static runtime and the compiler dynamic runtime (possibly different versions) in the same process. I'd expect that this would "just work", but this option exposes any issues in this to Open MPI. I'd vote for #1, but am curious about other's thoughts as well. |
|
Yeah, having a FWIW:
It might be sufficient to do option 1 (documentation in README/FAQ) and if it becomes a consistent problem, we can resurrect the idea of |
|
I think the option 1 is good, I have tested that the following steps are enough to resolve the old problem:
libopen-opal.so, libopen-rte.so and all orte binaries under bin will be linked to intel libs statically. Other files won't be touched. And libmpi.so will be linked to intel libs dynamically, but this is ok, user can use"-x LD_LIBRARY_PATH=..." to set the right path dynamically for MPI applications. |
|
i agree only that being said, i am not sure it is quite right to override LDFLAGS on the command line. on one hand, a at this stage, i'd rather do something like or and document this in a FAQ make sense ? |
|
Why is it bad to pass |
|
@jsquyres I have no strong reason to say it is bad :), but only want to keep it within a narrow scope, it should be less risky. Actually, I like this way, otherwise I need some time to learn autotools :) |
|
Ok, cool. I go back to what I said in #818 (comment), then. :-) |
|
@jsquyres I realized your idea is better than me after I did some works on the configure.ac, *.m4 and some Makefile.am. For example, I passed the "-Wc, -static-intel" to build libopen-opal.so, libopen-orte.so and orted successfully, but the testing showed that I also need pass it to mca_shmem_mmap.so, mca_shmem_posix and etc. So to narrow down the scope is good theoretically, but consider the maintenance, passing "-Wc, -static-intel" to all is better. Please give your comment :), thanks. |
|
I'm fine with either approach as described in #818 (comment) -- the documentation route is easier; the |
|
An Huang? Aren't you updating the FAQ? |
|
@gpaulsen Yes, I am learning the material about https://help.github.com/categories/collaborating and git, I need to be careful when I do this for the first time :) |
|
@jsquyres For this problem, I think we only need to add something into http://www.open-mpi.org/faq/?category=mpi-apps, what is the process for it? Thanks. |
|
@anhzhang Any progress on the FAQ? |
|
@jsquyres Sorry for being quiet for a long time because we have a very long National Day Holiday :) |
|
@anhzhang Hope you had a good vacation! You're close -- make the pull requests against the open-mpi/ompi-www repo. Then we can comment on it, etc. from there. |
This is the fix to #818, open-mpi/ompi#818
|
To close out this issue: we decided to make this an FAQ item and leave autogen/configure alone. The FAQ item has been added, so I'm closing this issue (and #829). |
…logical.endpoint.rank btl-portals4: set endpoint rank even if endpoint already exists
This is the fix to #818, open-mpi/ompi#818 Former-commit-id: 8ab0a42
This is the fix to #818, open-mpi/ompi#818 Former-commit-id: 8ab0a42
From #729:
Jeff Squyres recommend this approach to solve this type of issue:
I do like the idea of a
--enable-static-runtimekind of configure option. This would seem to solve all the problems, and not require the user to know/do anything at runtime.The only problem is: I'm not entirely sure how to do that.
--static-inteloption, but that doesn't make a wholly staticorted, for example (it just statically links in the Intel libraries -- not all libraries)....after thinking about this for a few minutes, it may be sufficient to do something like this:
Assuming
$(RUNTIME_LDFLAGS)is used when linking theorted(and any other relevant executables?), Libtool might automatically translate--staticinto whatever is relevant for the underlying compiler, and we only have one special case for the Intel compiler suite.The text was updated successfully, but these errors were encountered: