Incomplete link interface with -Wl,--as-needed with mpich if no symbol of deal.II is used

[tjhei]

I think there is something messed up with the cmake configuration that causes not all MPI libraries to be listed when we set up a test or auto pilot project. Everything works as expected if I use symbols from deal.II. But if my example.cc files is something like:

#include <mpi.h>
int main(int argc, char **argv)
{
  MPI_Init(&argc, &argv);
  int rank;
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  int size;
  MPI_Comm_size(MPI_COMM_WORLD, &size);
  MPI_Finalize();
}

I am getting linker errors:

/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libmpich.so: undefined reference to `MPL_trid'
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libmpich.so: undefined reference to `MPL_trfree'
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libmpich.so: undefined reference to `MPL_trvalid'
/usr/lib/gcc/x86_64-linux-gnu/4.6/../../../../lib/libmpich.so: undefined reference to `MPL_env2str'
[...]

because libmpl is not pulled in. When I look at FindMPI.cmake I can see that it should export MPI_LIBRARIES but it doesn't:

$ cmake -AL . | grep MPI_
MPI_CXX_COMPILER:FILEPATH=/usr/bin/c++
MPI_CXX_COMPILE_FLAGS:STRING=
MPI_CXX_INCLUDE_PATH:STRING=/usr/include/mpich2
MPI_CXX_LIBRARIES:STRING=/usr/lib/libmpichcxx.so;/usr/lib/libmpich.so;/usr/lib/libopa.so;/usr/lib/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthread.so
MPI_CXX_LINK_FLAGS:STRING= -Wl,-Bsymbolic-functions  -Wl,-z,relro
MPI_C_COMPILER:FILEPATH=/usr/bin/cc
MPI_C_COMPILE_FLAGS:STRING= -D_FORTIFY_SOURCE=2
MPI_C_INCLUDE_PATH:STRING=/usr/include/mpich2
MPI_C_LIBRARIES:STRING=/usr/lib/libmpich.so;/usr/lib/libopa.so;/usr/lib/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthread.so
MPI_C_LINK_FLAGS:STRING= -Wl,-Bsymbolic-functions  -Wl,-z,relro
MPI_EXTRA_LIBRARY:STRING=/usr/lib/libmpich.so;/usr/lib/libopa.so;/usr/lib/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthread.so
MPI_LIBRARY:FILEPATH=/usr/lib/libmpichcxx.so
MPI_MPI_H:FILEPATH=/usr/include/mpich2/mpi.h

Any thoughts on what is going on here? @tamiko ?

[drwells]

Yes, I just noticed this as well. I cannot even finish running cmake without commenting out

SET_IF_EMPTY(MPI_CXX_COMPILER ${CMAKE_CXX_COMPILER})
IF(CMAKE_C_COMPILER_WORKS)
  SET_IF_EMPTY(MPI_C_COMPILER ${CMAKE_C_COMPILER}) # for good measure
ELSE()
  MESSAGE(STATUS
    "No suitable C compiler was found! MPI C interface can not be "
    "autodetected"
    )
ENDIF()
IF(CMAKE_Fortran_COMPILER_WORKS)
  SET_IF_EMPTY(MPI_Fortran_COMPILER ${CMAKE_Fortran_COMPILER}) # for good measure
ELSE()
  MESSAGE(STATUS
    "No suitable Fortran compiler was found! MPI Fortran interface can "
    "not be autodetected"
    )
ENDIF()

with a very new version of CMake (3.4.0) and OpenMPI. Things seem to work on the other computer I have handy, which has 2.8.11 and MPICH.

@tjhei What version of CMake did you use?

[tjhei]

I cannot even finish running cmake without commenting out

Well, that is a different issue, because configuring and compiling deal.II and examples works just fine here. Just compiling a test that doesn't directly use deal.II (weird, I know) fails.

@tjhei What version of CMake did you use?

3.3.0

[tamiko]

[...] messed up with the cmake configuration that causes not all MPI libraries to be listed

I don't think so. We populate MPI_CXX_LIBRARIES with all C++, Fortran and C support libraries and in your case it lists libmpl:

MPI_CXX_LIBRARIES:STRING=/usr/lib/libmpichcxx.so;/usr/lib/libmpich.so;/usr/lib/libopa.so;/usr/lib/libmpl.so;/usr/lib/x86_64-linux-gnu/librt.so;/usr/lib/libcr.so;/usr/lib/x86_64-linux-gnu/libpthre

The problem is the simple fact that within the build system the targets are linked with -Wl,-as-needed and mpich does not like that [1].

[1] http://lists.mcs.anl.gov/pipermail/mpich-discuss/2010-November/008389.html

[tjhei]

okay, any suggestion how I can fix this at least on my machine? Can I stuff --no-is-needed into MPI_CXX_LINKER_FLAGS during deal.II configure?

[tamiko]

okay, any suggestion how I can fix this at least on my machine? Can I
stuff --no-is-needed into MPI_CXX_LINKER_FLAGS during deal.II
configure?

You can just configure with

 -DDEAL_II_HAVE_FLAG_Wl__as_needed=false

Is this for one of the quicktests? If, yes. Does it help to force
external linkage against libdeal_II.so? Alternatively, we can override
-WL,-as-needed for this test and/or start to distinguish between library
and executable linker flags (which might be overkill at the moment).

[tjhei]

You can just configure with

 -DDEAL_II_HAVE_FLAG_Wl__as_needed=false

doesn't work and neither does adding -Wl--no-as-needed to DEAL_II_LINKER_FLAGS.

Is this for one of the quicktests?

No, I was using the deal.II CMakeList.txt project for a a simple c++ test application so I didn't have to bother with c++11/MPI/boost setup. And as_needed will rip out libmpl if I don't use anything from deal.II itself. :-)

But this also failed if I create a normal test that does not reference anything from deal.II.

[tamiko]

@tjhei Can you post the relevant bits of detailed.log and tell me on which architecture with what compiler and version of mpich you encounter this? I'll try to reproduce.

[tjhei]

Correction, if I set DEAL_II_LINKER_FLAGS to -Wl,--no-as-needed (without forgetting the comma), it works. Do you still think -DDEAL_II_HAVE_FLAG_Wl__as_needed=false should work? Should I try again?

heister@timo ~ $ uname -a
Linux timo 3.13.0-66-generic #108~precise1-Ubuntu SMP Thu Oct 8 10:07:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux
heister@timo ~ $ lsb_release -a
LSB Version:    core-2.0-amd64:core-2.0-noarch:core-3.0-amd64:core-3.0-noarch:core-3.1-amd64:core-3.1-noarch:core-3.2-amd64:core-3.2-noarch:core-4.0-amd64:core-4.0-noarch
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:    12.04
Codename:   precise
heister@timo ~ $ mpicxx -v
mpicxx for MPICH2 version 1.4.1
Using built-in specs.
COLLECT_GCC=c++
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/4.6/lto-wrapper
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu/Linaro 4.6.3-1ubuntu5' --with-bugurl=file:///usr/share/doc/gcc-4.6/README.Bugs --enable-languages=c,c++,fortran,objc,obj-c++ --prefix=/usr --program-suffix=-4.6 --enable-shared --enable-linker-build-id --with-system-zlib --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --with-gxx-include-dir=/usr/include/c++/4.6 --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --enable-gnu-unique-object --enable-plugin --enable-objc-gc --disable-werror --with-arch-32=i686 --with-tune=generic --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) 

[tamiko]

Do you still think -DDEAL_II_HAVE_FLAG_Wl__as_needed=false should work? Should I try again?

Maybe you hit a "three state". If you set -DDEAL_II_HAVE_FLAG_Wl__as_needed=false in CMakeCache.txt the corresponding entry should be set to false:

DEAL_II_HAVE_FLAG_Wl__as_needed:INTERNAL=0

But this only means that we do not force -Wl,--as-needed any more. Your linux distribution, or gcc, or binutils might have changed the default behavior to -Wl,-as-needed. In this case you have to explicitly disable it with -Wl,--no-as-needed.

[tamiko]

@tjhei Is this still a problem?

[tamiko]

:-)

[tjhei]

Yes, I can still reproduce this problem.

[tjhei]

I realized I can set CMAKE_EXE_LINKER_FLAGS to -Wl,--no-as-needed in the project and link the program successfully. While not great, I don't see a way to fix this inside deal.II. Do you?

[tamiko]

@tjhei DEAL_II_LINKER_FLAGS should work as well, doesn't it?

No, I don't see a nice workaround. Simply not setting -Wl,--as-needed is not an option - a number of major Linux distributions set as-needed per default. Explicitly compiling with -Wl,--no-as-needed is equally ugly. And, as a matter of fact, we want to have -Wl,--as-needed...

[tjhei]

@tjhei DEAL_II_LINKER_FLAGS should work as well, doesn't it?

That would be something I need to do when configuring deal.II. Yes, that worked back then.

[drwells]

I don't think that this is still an issue - if it is we should reopen.