New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compiler Wrappers Erroneously in Path for `spack setup`; breaks things sometimes. #2055

Open
pramodk opened this Issue Oct 19, 2016 · 5 comments

Comments

Projects
None yet
2 participants
@pramodk
Collaborator

pramodk commented Oct 19, 2016

I am following Developing Software with Spack section from documentation and see following error:

$ mkdir build; cd build
$ ../spconfig.py ..
-- The C compiler identification is GNU 4.9.3
-- The CXX compiler identification is GNU 4.9.3
-- Checking whether C compiler has -isysroot
-- Checking whether C compiler has -isysroot - yes
-- Checking whether C compiler supports OSX deployment target flag
-- Checking whether C compiler supports OSX deployment target flag - yes
-- Check for working C compiler: /Users/kumbhar/workarena/software/sources/spack/opt/spack/darwin-elcapitan-x86_64/gcc-4.2.1/gcc-4.9.3-3yghgmzvmbiqmem7lpu2dmcqz6qoqcjd/bin/gcc
-- Check for working C compiler: /Users/kumbhar/workarena/software/sources/spack/opt/spack/darwin-elcapitan-x86_64/gcc-4.2.1/gcc-4.9.3-3yghgmzvmbiqmem7lpu2dmcqz6qoqcjd/bin/gcc -- broken
….
  /Users/kumbhar/workarena/software/sources/spack/opt/spack/darwin-elcapitan-x86_64/gcc-4.2.1/gcc-4.9.3-3yghgmzvmbiqmem7lpu2dmcqz6qoqcjd/bin/gcc
  -Wl,-search_paths_first -Wl,-headerpad_max_install_names
  CMakeFiles/cmTC_afeb5.dir/testCCompiler.c.o -o cmTC_afeb5

  Spack compiler must be run from Spack! Input 'SPACK_PREFIX' is
  missing.collect2: error: ld returned 1 exit status

In some other thread (#1111) I came across spack env. Using this I get all environment as:

spack env mylib@develop bash

and then it works fine. Is above the missing step from the workflow document?

I have another question related to extending CMakePackage and spconfig.py:

When I started writing packages for CMake based projects, I have extended packages from Package. Now when I do spack setup my_spec in the source directory of the project, spconfig.py is not generated. But I have install method defined in the package.py with spack-build as build directory. So when I invokespack setup, it build and install the package. And then for development purpose I just do:

cd speck-build
spack env spec_with_spaces - -  bash  

# change code - make - change code .....

which works fine! Is there any drawback of this compared to having spconfig.py?

@citibeth : any suggestion?

@citibeth

This comment has been minimized.

Show comment
Hide comment
@citibeth

citibeth Oct 20, 2016

Member

A bit of background... for a variety of reasons, Spack wraps your compiler in shell-script wrappers, designed to be run only from within Spack. The spconfig.py script is supposed to do exactly the same build as what Spack would do if you called spack install, EXCEPT that it calls the compiler directly, rather than Spack's wrappers (which only work from within Spack). On my system, that's how it's always worked for me. I have no idea why spconfig.py is trying to call the compiler wrappers in your case, maybe @tgamblin has some ideas. Can you report on the following to try to debug?

  1. Share your package.py, spconfig.py, packages.yaml and compilers.yaml.
  2. Do you get the same behavior building other stuff with spack setup, i.e. is this an issue with your particular package? Try building the ibmisc package with spack setup, which others have had success with.
  3. Are you able to try the same build on a non-Mac?

It looks like the spack env is a workaround that makes the Spack compiler wrappers "work" even from outside of Spack?


When I started writing packages for CMake based projects, I have extended packages from Package

Yes... spack setup needs to get CMake arguments without running the instlall. So the install() method was broken up into phases to enable this. Hence the need to subclass from CMakePackage. spack setup will not work if you subclass from Package directly.

Is there any drawback of this compared to having spconfig.py?

Yes:

  1. The behavior of spack setup on packages subclassed directly from Package is undefined.
  2. You have to run a new shell with a particular environment in order to build. That is a little more cumbersome than having the required environment in spconfig.py
  3. Depending on the complexity of your project, running Spack can be really slow. The spconfig.py approach minimizes the number of times Spack must be run.

That said... I was not aware of the possibilities you brought up, thank you for mentioning them!

Member

citibeth commented Oct 20, 2016

A bit of background... for a variety of reasons, Spack wraps your compiler in shell-script wrappers, designed to be run only from within Spack. The spconfig.py script is supposed to do exactly the same build as what Spack would do if you called spack install, EXCEPT that it calls the compiler directly, rather than Spack's wrappers (which only work from within Spack). On my system, that's how it's always worked for me. I have no idea why spconfig.py is trying to call the compiler wrappers in your case, maybe @tgamblin has some ideas. Can you report on the following to try to debug?

  1. Share your package.py, spconfig.py, packages.yaml and compilers.yaml.
  2. Do you get the same behavior building other stuff with spack setup, i.e. is this an issue with your particular package? Try building the ibmisc package with spack setup, which others have had success with.
  3. Are you able to try the same build on a non-Mac?

It looks like the spack env is a workaround that makes the Spack compiler wrappers "work" even from outside of Spack?


When I started writing packages for CMake based projects, I have extended packages from Package

Yes... spack setup needs to get CMake arguments without running the instlall. So the install() method was broken up into phases to enable this. Hence the need to subclass from CMakePackage. spack setup will not work if you subclass from Package directly.

Is there any drawback of this compared to having spconfig.py?

Yes:

  1. The behavior of spack setup on packages subclassed directly from Package is undefined.
  2. You have to run a new shell with a particular environment in order to build. That is a little more cumbersome than having the required environment in spconfig.py
  3. Depending on the complexity of your project, running Spack can be really slow. The spconfig.py approach minimizes the number of times Spack must be run.

That said... I was not aware of the possibilities you brought up, thank you for mentioning them!

@citibeth

This comment has been minimized.

Show comment
Hide comment
@citibeth

citibeth Oct 22, 2016

Member

Good news, I ran into this problem too. But only on SOME computers... I don't yet know why some but not all. Anyway... look in the generated spconfig.py files, I see the following:

env['PATH'] = ":".join(cmdlist("""
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.1-xfzruydrrrsee7v264k7sjlzxtsvedhg/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/python-3.5.2-d5igtz5ukjiqtaxlfyror562fvygfiv4/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/netcdf-cxx4-4.3.0-6sb6flocof7yu2jis2jyhp3xhc2yzg4d/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/py-numpy-1.11.1-lhvrnmqb3f6phgnd4gs2pgqo562elian/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/udunits2-2.2.20-rvmcxyelfy6yhao4g4xdki4tts6numbe/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/proj-4.9.2-f654335yo4t4uz3fdd67vytqnmvo5n4p/bin
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/case-insensitive
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/gcc
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/binutils-2.27-vdugmzyoeozobcy2zppn6kjpyyorjfef/bin
    /home/rpfische/git/modele-control/bin
    /usr/local/other/SLES11.3/openmpi/1.10.1/gcc-5.3/bin
    /usr/local/other/SLES11.3/gcc/5.3.0/bin
    /usr/local/other/SLES11.3/git/2.7.4/libexec/git-core
    /usr/local/other/SLES11.3/git/2.7.4/bin
    /usr/local/other/SLES11.3/asciidoc/8.6.8/bin
    /home/rpfische/spack3/bin
    /usr/local/bin
    /usr/bin
    /bin
    /usr/bin/X11
    /usr/X11R6/bin
    /usr/games
    /opt/kde3/bin
    /opt/ibutils/bin
    /usr/lib/mit/bin
    /usr/lib/mit/sbin
    /usr/slurm/bin
    /home/rpfische/spack2/bin
"""))

The compiler wrappers are found in the following locations:

    /gpfsm/dnb53/rpfische/spack3/lib/spack/env
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/case-insensitive
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/gcc

These come BEFORE the real compiler, which is found at:

    /usr/local/other/SLES11.3/gcc/5.3.0/bin

@tgamblin @alalazo
The workaround for now is to remove the three compiler wrapper lines from spconfig.py, then it works (for me) like a charm!

The long-term solution? I could certainly add some code to make sure those three lines don't make it into spconfigpy. But I'm wondering if this is related to commit e9d4780 on Oct 4. I'd like to get a second opinion on what's going on here.

e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 263)     # These variables control compiler wrapper be
havior
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 264)     env.set_path(SPACK_RPATH_DEPS, [d.prefix for 
d in get_rpath_deps(pkg)])
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 265)     env.set_path(SPACK_LINK_DEPS, [
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 266)         d.prefix for d in pkg.spec.traverse(root=
False, deptype=('link'))])
Member

citibeth commented Oct 22, 2016

Good news, I ran into this problem too. But only on SOME computers... I don't yet know why some but not all. Anyway... look in the generated spconfig.py files, I see the following:

env['PATH'] = ":".join(cmdlist("""
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/cmake-3.6.1-xfzruydrrrsee7v264k7sjlzxtsvedhg/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/python-3.5.2-d5igtz5ukjiqtaxlfyror562fvygfiv4/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/netcdf-cxx4-4.3.0-6sb6flocof7yu2jis2jyhp3xhc2yzg4d/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/py-numpy-1.11.1-lhvrnmqb3f6phgnd4gs2pgqo562elian/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/udunits2-2.2.20-rvmcxyelfy6yhao4g4xdki4tts6numbe/bin
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/proj-4.9.2-f654335yo4t4uz3fdd67vytqnmvo5n4p/bin
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/case-insensitive
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/gcc
    /gpfsm/dnb53/rpfische/spack3/opt/spack/linux-SuSE11-x86_64/gcc-5.3.0/binutils-2.27-vdugmzyoeozobcy2zppn6kjpyyorjfef/bin
    /home/rpfische/git/modele-control/bin
    /usr/local/other/SLES11.3/openmpi/1.10.1/gcc-5.3/bin
    /usr/local/other/SLES11.3/gcc/5.3.0/bin
    /usr/local/other/SLES11.3/git/2.7.4/libexec/git-core
    /usr/local/other/SLES11.3/git/2.7.4/bin
    /usr/local/other/SLES11.3/asciidoc/8.6.8/bin
    /home/rpfische/spack3/bin
    /usr/local/bin
    /usr/bin
    /bin
    /usr/bin/X11
    /usr/X11R6/bin
    /usr/games
    /opt/kde3/bin
    /opt/ibutils/bin
    /usr/lib/mit/bin
    /usr/lib/mit/sbin
    /usr/slurm/bin
    /home/rpfische/spack2/bin
"""))

The compiler wrappers are found in the following locations:

    /gpfsm/dnb53/rpfische/spack3/lib/spack/env
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/case-insensitive
    /gpfsm/dnb53/rpfische/spack3/lib/spack/env/gcc

These come BEFORE the real compiler, which is found at:

    /usr/local/other/SLES11.3/gcc/5.3.0/bin

@tgamblin @alalazo
The workaround for now is to remove the three compiler wrapper lines from spconfig.py, then it works (for me) like a charm!

The long-term solution? I could certainly add some code to make sure those three lines don't make it into spconfigpy. But I'm wondering if this is related to commit e9d4780 on Oct 4. I'd like to get a second opinion on what's going on here.

e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 263)     # These variables control compiler wrapper be
havior
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 264)     env.set_path(SPACK_RPATH_DEPS, [d.prefix for 
d in get_rpath_deps(pkg)])
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 265)     env.set_path(SPACK_LINK_DEPS, [
e9d4780b (Todd Gamblin       2016-10-04 09:40:28 -0700 266)         d.prefix for d in pkg.spec.traverse(root=
False, deptype=('link'))])
@citibeth

This comment has been minimized.

Show comment
Hide comment
@citibeth

citibeth Oct 22, 2016

Member

OK, I don't know why it's not working. Later on in spconfig.py is:

    -DCMAKE_C_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/gcc
    -DCMAKE_CXX_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/g++
    -DCMAKE_Fortran_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran

On the computer where it does work, these lines seem to be causing CMake to pick up the real compiler, not the Spack wrappers. The spconfig.py is essentially the same on the computer that doesn't work; but there, CMake picks up the Spack wrappers instead. In BOTH cases, the compiler is NOT in a system location. On the working computer, it's Spack-built; on the non-working computer it's handbuilt and requires a module to run properly.

Member

citibeth commented Oct 22, 2016

OK, I don't know why it's not working. Later on in spconfig.py is:

    -DCMAKE_C_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/gcc
    -DCMAKE_CXX_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/g++
    -DCMAKE_Fortran_COMPILER=/usr/local/other/SLES11.3/gcc/5.3.0/bin/gfortran

On the computer where it does work, these lines seem to be causing CMake to pick up the real compiler, not the Spack wrappers. The spconfig.py is essentially the same on the computer that doesn't work; but there, CMake picks up the Spack wrappers instead. In BOTH cases, the compiler is NOT in a system location. On the working computer, it's Spack-built; on the non-working computer it's handbuilt and requires a module to run properly.

@citibeth citibeth changed the title from Development workflow : spack setup need spack env ? to Compiler Wrappers Erroneously in Path for `spack setup`; breaks things sometimes. Oct 23, 2016

@citibeth citibeth added the bug label Oct 23, 2016

@citibeth citibeth self-assigned this Oct 23, 2016

@pramodk

This comment has been minimized.

Show comment
Hide comment
@pramodk

pramodk Oct 23, 2016

Collaborator

Thanks @citibeth for detailed information! I don't have my dev environment setup at the moment, will provide my configurations soon.

Here are some additional points:

  • I have packages inheriting Package class. When I do spack setup, it install the packages. But as build directory is created in the source, I can use it after spack env.
  • For autotools based projects this works fine i.e. spack setup build and install the package. From build directory I can just do make/make install without spack env.
Collaborator

pramodk commented Oct 23, 2016

Thanks @citibeth for detailed information! I don't have my dev environment setup at the moment, will provide my configurations soon.

Here are some additional points:

  • I have packages inheriting Package class. When I do spack setup, it install the packages. But as build directory is created in the source, I can use it after spack env.
  • For autotools based projects this works fine i.e. spack setup build and install the package. From build directory I can just do make/make install without spack env.
@citibeth

This comment has been minimized.

Show comment
Hide comment
@citibeth

citibeth Oct 25, 2016

Member

I think I found the problem, don't think we need additional details.

On Sun, Oct 23, 2016 at 9:48 AM, Pramod Kumbhar notifications@github.com
wrote:

Thanks @citibeth https://github.com/citibeth for detailed information!
I don't have my dev environment setup at the moment, will provide my
configurations soon.

Here are some additional points:

  • I have packages inheriting Package class. When I do spack setup, it
    install the packages. But as build directory is created in the source,
    I can use it after spack env.
  • For autotools based projects this works fine i.e. spack setup build
    and install the package. From build directory I can just do make/make
    install without spack env.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2055 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd7rNdt7Rd-g8MYIYue6eRww4hVRjks5q22WqgaJpZM4KbW-j
.

Member

citibeth commented Oct 25, 2016

I think I found the problem, don't think we need additional details.

On Sun, Oct 23, 2016 at 9:48 AM, Pramod Kumbhar notifications@github.com
wrote:

Thanks @citibeth https://github.com/citibeth for detailed information!
I don't have my dev environment setup at the moment, will provide my
configurations soon.

Here are some additional points:

  • I have packages inheriting Package class. When I do spack setup, it
    install the packages. But as build directory is created in the source,
    I can use it after spack env.
  • For autotools based projects this works fine i.e. spack setup build
    and install the package. From build directory I can just do make/make
    install without spack env.


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#2055 (comment), or mute
the thread
https://github.com/notifications/unsubscribe-auth/AB1cd7rNdt7Rd-g8MYIYue6eRww4hVRjks5q22WqgaJpZM4KbW-j
.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment