Problem building with module-loaded compiler

[spakin]

I'm trying out spack for the first time and ran into trouble when I tried to install a package after loading a compiler with the module command:

$ gcc --version | head -1
gcc (GCC) 4.4.7 20120313 (Red Hat 4.4.7-4)
$ module show compilers/gcc/4.8.3
-------------------------------------------------------------------
/etc/modulefiles/compilers/gcc/4.8.3:

module-whatis    load GCC 4.8.3 environment 
module-whatis    Modifies: PATH, LD_LIBRARY_PATH, MANPATH 
prepend-path     PATH /projects/opt/gcc/4.8.3/bin 
prepend-path     LD_LIBRARY_PATH /projects/opt/gcc/4.8.3/lib64:/projects/opt/gcc/4.8.3/lib 
prepend-path     MANPATH /projects/opt/gcc/4.8.3/share/man 
setenv       CC gcc 
setenv       CXX g++ 
setenv       CPP cpp 
setenv       FC gfortran 
setenv       F77 gfortran 
setenv       F90 gfortran 
-------------------------------------------------------------------

$ module load compilers/gcc/4.8.3
$ gcc --version | head -1
gcc (GCC) 4.8.3
$ spack compilers
==> Available compilers
-- gcc ----------------------------------------------------------
gcc@4.4.7
$ spack compiler add
==> Added 1 new compiler to /home/pakin/.spackconfig
gcc@4.8.3
$ spack install libelf
==> Installing libelf
==> Trying to fetch from http://www.mr511.de/software/libelf-0.8.12.tar.gz
######################################################################## 100.0%
==> Staging archive: /home/pakin/spack/var/spack/stage/libelf@0.8.12%gcc@4.8.3=unknown_arch/libelf-0.8.12.tar.gz
==> Created stage in /home/pakin/spack/var/spack/stage/libelf@0.8.12%gcc@4.8.3=unknown_arch.
==> No patches needed for libelf.
==> Building libelf.
creating cache ./config.cache
checking whether make sets ${MAKE}... yes
checking for gcc... cc
checking whether the C compiler (cc  ) works... no
configure: error: installation or configuration problem: C compiler cannot create executables.
Traceback (most recent call last):
  File "/home/pakin/spack/lib/spack/spack/package.py", line 754, in do_install
self.install(self.spec, self.prefix)
  File "/home/pakin/spack/var/spack/packages/libelf/package.py", line 43, in install
"--disable-debug")
  File "/home/pakin/spack/lib/spack/spack/util/executable.py", line 83, in __call__
% (" ".join(cmd), proc.returncode))
spack.util.executable.ProcessError: command './configure --prefix=/home/pakin/spack/opt/unknown_arch/gcc@4.8.3/libelf@0.8.12 --enable-shared --disable-dependency-tracking --disable-debug' returned error code 1
$ cat /tmp/spack-stage/spack-stage-J87_JC/libelf-0.8.12/config.log
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

configure:582: checking whether make sets ${MAKE}
configure:611: checking for gcc
configure:724: checking whether the C compiler (cc  ) works
configure:740: cc -o conftest    conftest.c  1>&5
/projects/opt/gcc/4.8.3/packages/gcc-4.8.3/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.8.3/cc1: error while loading shared libraries: libmpc.so.3: cannot open shared object file: No such file or directory
configure: failed program was:

#line 735 "configure"
#include "confdefs.h"

main(){return(0);}
$ ldd /projects/opt/gcc/4.8.3/packages/gcc-4.8.3/bin/../libexec/gcc/x86_64-unknown-linux-gnu/4.8.3/cc1
    linux-vdso.so.1 =>  (0x00007fff5b9ff000)
    libmpc.so.3 => /projects/opt/gcc/4.8.3/lib/libmpc.so.3 (0x00002ad3b51d7000)
    libmpfr.so.4 => /projects/opt/gcc/4.8.3/lib/libmpfr.so.4 (0x00002ad3b53ef000)
    libgmp.so.10 => /projects/opt/gcc/4.8.3/lib/libgmp.so.10 (0x00002ad3b564b000)
    libdl.so.2 => /lib64/libdl.so.2 (0x0000003167000000)
    libz.so.1 => /lib64/libz.so.1 (0x0000003167400000)
    libstdc++.so.6 => /projects/opt/gcc/4.8.3/lib64/libstdc++.so.6 (0x00002ad3b58e3000)
    libm.so.6 => /lib64/libm.so.6 (0x0000003166800000)
    libgcc_s.so.1 => /projects/opt/gcc/4.8.3/lib64/libgcc_s.so.1 (0x00002ad3b5bed000)
    libc.so.6 => /lib64/libc.so.6 (0x0000003166400000)
    /lib64/ld-linux-x86-64.so.2 (0x0000003166000000)

It looks like spack is discarding my LD_LIBRARY_PATH. Is there a way to tell it not to?

— Scott

[tgamblin]

So, this is interesting. All the LLNL custom gcc installs use either static dependency libraries, or they rely on the system dependency libraries (libmpfr, etc). The idea with Spack is that things like LD_LIBRARY_PATH should not matter, and moreover that they shouldn't interfere with the package build. I blow away LD_LIBRARY_PATH and populate it with /lib and /lib64 locations only of dependencies during a spack build. I also add RPATHs for all the dependencies to the compile line. If I include your LD_LIBRARY_PATH, I run the risk of polluting the package build with gcc's dependencies. What if gcc itself builds a binary that depends on mpfr?

For example, when Spack builds gcc, it actually bakes in the dependency RPATHs using its cc wrapper.

I can't do that with external compilers, but I definitely want to support external compilers. Would it be reasonable to record the LD_LIBRARY_PATH on "spack compiler add" and to use that particular one to invoke the real gcc from within spack's cc? That would keep LD_LIBRARY_PATH out of the build environment and out of test runs (e.g., in configure) during the build. When the build invokes Spack's cc, it makes sure to use the LD_LIBRARY_PATH needed by the compiler.

Does that seem reasonable?

[spakin]

Wouldn't it be sufficient simply to prepend Spack's dependencies before the existing LD_LIBRARY_PATH? That way, if a dependency can be satisfied by Spack it will be. In cases like mine, the external compiler should still be able to run.

I don't know how Spack decides what to include in each generated executable's RPATH, but I'd think that properly ordering LD_LIBRARY_PATH would just do the right thing.

— Scott

[tgamblin]

Not exactly -- if I do as you suggest, what happens when I unload the module? The user should still be able to build thing with, say, %gcc@4.8.3 and have it work. If I rely on modules, it won't. I could simplify implementation by putting compiler LD_LIBRARY_PATHs last in the environment and not bothering with passing them to wrappers to be set only when compilers run, but I still think I have to remember the LD_LIBRARY_PATH per compiler...

[AdrianRodriguezVilas]

Hi, I'm just posting this comment because I found the same problem and I'm interested in knowing if this is already solved somehow or is planned to be in a future version.

@tgamblin Your solution seems reasonable to me. I understand that solving it that way would allow to load/unload the compiler module without affecting to the builds made with it previously, am I right?

[adamjstewart]

This issue is still being reported:

https://groups.google.com/forum/#!topic/spack/I7CQgmwHmLI

[adamjstewart]

Also see #332.

[tgamblin]

Fixed by #2279.

[junghans]

/cc @louisvernon @rspavel

[scheibelp]

@junghans could you expand on #6 (comment)? For example if you are getting the exact same error as #6 (comment) it would be good to know that (along with additional details e.g. relevant entries from compilers.yaml or packages.yaml) or if this came up in another issue if that could be mentioned here (at the present time #5705 isn't confirmed to be related to compiler modules AFAIK).

[louisvernon]

This was reopened by @junghans as we unclear on how #2279 fixes the originally described issue. What is the current expected behavior for a compiler which requires the setting of LD_LIBRARY_PATH? If the expectation is to explicitly manage the environment through customization of compilers.yaml, how does this relate to #2279?

[scheibelp]

@louisvernon thanks for the extra info.

@tgamblin FYI

What is the current expected behavior for a compiler which requires the setting of LD_LIBRARY_PATH?

Prior to #5599 (merged last week) LD_LIBRARY_PATH as set by the compiler module would not have been respected. Now it should be - @becker33 does that conflict with #6 (comment)?

If the expectation is to explicitly manage the environment through customization of compilers.yaml...

Sorry I'm lacking the context for this one. Is this related to desired behavior not yet available or expected behavior that is not working?

[becker33]

Closing as outdated

[mohit9638iitk]

Spack install can't find shared libraries when using intel compilers