v1.10 build failing on some platforms with link error: missing -pthread

[jsquyres]

On RHEL 6.5, it looks like the addition of

OPAL_SEARCH_LIBS_CORE([clock_gettime], [rt])

in open-mpi/ompi-release#1181 caused -pthread to disappear from CFLAGS ( found this via git bisect). This PR was merged after the nightly v1.10 snapshot tarball was made last night, so it didn't show up in MTT results from last night.

Specifically, before this commit, config.log shows:

configure:63325: checking if C compiler and POSIX threads work as is
configure:63372: gcc -o conftest -g -Wall -Wundef -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic -Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing   conftest.c -lm -lutil  >&5
/tmp/ccRfWycq.o: In function `main':
conftest.c:(.text+0xb6): undefined reference to `__pthread_register_cancel'

and then configure later adds -pthread to CFLAGS. This seems to be the correct course of action.

But after that commit, config.log shows:

configure:63545: checking if C compiler and POSIX threads work as is
configure:63592: gcc -o conftest -g -Wall -Wundef -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic -Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing   conftest.c -lrt -lm -lutil  >&5
configure:63592: $? = 0
configure:63608: result: yes

I.e., configure concludes that it does not need any additional CFLAGS to compile pthreads things, so it doesn't add -pthread to CFLAGS.

The only difference between the two test commands is the addition of -lrt, which came from the OPAL_SEARCH_LIBS_CORE addition to configure.ac.

---

I did not follow the entire cause-and-effect to see what exactly is happening, but I can see the following difference in the resulting builds:

Before (i.e., works properly):

libtool: link: gcc -g -Wall -Wundef -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic -Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing -pthread -o .libs/opal_wrapper opal_wrapper.o  ../../../opal/.libs/libopen-pal.so -lnuma -ldl -lrt -lm -lutil -pthread -Wl,-rpath -Wl,/home/jsquyres/bogus/lib

After (i.e., fails with pthread_atfork missing symbol):

libtool: link: gcc -g -Wall -Wundef -Wno-long-long -Wsign-compare -Wmissing-prototypes -Wstrict-prototypes -Wcomment -pedantic -Werror-implicit-function-declaration -finline-functions -fno-strict-aliasing -o .libs/opal_wrapper opal_wrapper.o  ../../../opal/.libs/libopen-pal.so -lnuma -ldl -lrt -lm -lutil -Wl,-rpath -Wl,/home/jsquyres/bogus/lib

The only difference between those two is the missing -pthread in the second (bad) one.

This may not be happening on all platforms -- e.g., @rhc54 has cited that it is not happening for him on CentOS 7. I'm guessing that the pthread libraries are getting pulled in indirectly somehow on those platforms...?

[jsquyres]

I'm afraid I don't have cycles to look into this -- who has a few free cycles to chase this down?

[bosilca]

pthread is pulled automatically by -lrt as there is a dependence (at least on RedHat). This might explain why -lpthread disappear from the linking flags.

[rhc54]

so maybe someone with an affected system could please check adding -lpthread back to the CFLAGS after that check? my system works, and so this isn't something i can check

[jsquyres]

@bosilca FWIW: it was sorta related to that. What you describe explains why the problem doesn't happen on RHEL7. But it doesn't explain why configure was happy with its configure tests with just -lrt on RHEL6, but then it wasn't happy when compiling opal_wrapper (with just -lrt and no other pthread flags).

The real fix was on master (9d9d05b): @ggouaillardet added an explicit test for pthread_atfork into our pthread configury tests, which apparently makes them a bit more stringent and makes sure that something like -lrt doesn't cause a false positive (i.e., -lrt apparently pulls in some pthreads infrastructure, but not the bits that include pthread_atfork).

[ggouaillardet]

fwiw, i noted yet an other "librt crazyness" with Fujitsu compilers.
-lrt is appended under the hood when building a binary, but not when building a shared library.
as a consequence, Open MPI does not find any need for -lrt and components cannot be dlopen'ed.
as a workaround, i configure with --disable-dlopen and everything is fine.