Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Install openmpi with NAG compiler #15776

Open
ajkotobi opened this issue Mar 31, 2020 · 28 comments · Fixed by #15902
Open

Install openmpi with NAG compiler #15776

ajkotobi opened this issue Mar 31, 2020 · 28 comments · Fixed by #15902

Comments

@ajkotobi
Copy link
Contributor

Spack version

0.13.3

Steps to reproduce the issue

spack install -v openmpi@4.0.2 %nag

==> Installing libsigsegv
==> Searching for binary cache of libsigsegv
==> Finding buildcaches in /.../spack-mirrors/build_cache
==> No binary for libsigsegv found: installing from source
==> Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]
==> Error: TypeError: expected string or buffer

/.../lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None

Platform and user environment

Linux 2.6.32-754.23.1.el6.x86_64 #1 SMP Tue Sep 17 09:46:55 EDT 2019 x86_64 x86_64 x86_64 GNU/Linux
Description:	Red Hat Enterprise Linux Server release 6.10 (Santiago)
  • compilers.yaml
- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gcc
      cxx: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/g++
      f77: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gfortran
      fc: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gfortran
    operating_system: rhel6
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
    flags:
      cflags: -O2 -g -march=native
      fflags: -O2 -g -march=native
      cxxflags: -O2 -g -march=native
    spec: gcc@7.4.0
- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
      cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
      f77: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
      fc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
    operating_system: rhel6
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
    flags:
      cflags: -O2 -g -march=native
      fflags: -O2 -g -march=native
      cxxflags: -O2 -g -march=native
    spec: gcc@9.1.0
- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
      cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
      f77: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
      fc: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
    operating_system: rhel6
    target: x86_64
    flags:
      fflags: -mismatch
    spec: nag@7.0

Additional information

 spack spec --install-status openmpi@4.0.1 %nag
Input spec
--------------------------------
 -   openmpi@4.0.1%nag

Concretized
--------------------------------
 -   openmpi@4.0.1%nag@7.0 fflags="-mismatch" ~cuda+cxx_exceptions fabrics=none ~java~legacylaunchers~memchecker patches=073477a76bba780c67c36e959cd3ee6910743e2735c7e76850ffba6791d498e4 ~pmi schedulers=none ~sqlite3~thread_multiple+vt arch=linux-rhel6-haswell
 -       ^hwloc@1.11.11%nag@7.0 fflags="-mismatch" ~cairo~cuda~gl+libxml2~nvml+pci+shared arch=linux-rhel6-haswell
 -           ^libpciaccess@0.13.5%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^libtool@2.4.6%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -                   ^m4@1.4.18%nag@7.0 fflags="-mismatch"  patches=3877ab548f88597ab2327a2230ee048d2d07ace1062efe81fc92e91b7f39cd00,fc9b61654a3ba1a8d6cd78ce087e7c96366c290bc8d2c299f09828d793b853c8 +sigsegv arch=linux-rhel6-haswell
 -                       ^libsigsegv@2.12%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^pkgconf@1.6.3%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^util-macros@1.19.1%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -           ^libxml2@2.9.9%nag@7.0 fflags="-mismatch" ~python arch=linux-rhel6-haswell
 -               ^libiconv@1.16%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^xz@5.2.4%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^zlib@1.2.11%nag@7.0 fflags="-mismatch" +optimize+pic+shared arch=linux-rhel6-haswell
 -           ^numactl@2.0.12%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -               ^autoconf@2.69%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -                   ^perl@5.30.0%nag@7.0 fflags="-mismatch" +cpanm+shared+threads arch=linux-rhel6-haswell
 -                       ^gdbm@1.18.1%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -                           ^readline@8.0%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell
 -                               ^ncurses@6.1%nag@7.0 fflags="-mismatch" ~symlinks~termlib arch=linux-rhel6-haswell
 -               ^automake@1.16.1%nag@7.0 fflags="-mismatch"  arch=linux-rhel6-haswell

During installation of openmpi@4.0.2 %nag on the first dependency it shows

Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]

Any idea where is it going wrong?

Thanks

@adamjstewart
Copy link
Member

The warning is expected, the error is not. Can you run with spack -d install?

@ajkotobi
Copy link
Contributor Author

Here is the output


==> [2020-03-31-18:08:57.696735] Installing libsigsegv
==> [2020-03-31-18:08:57.696817] Searching for binary cache of libsigsegv
==> [2020-03-31-18:08:57.698469] Reading config file /mnt/lustre01/sw/sw-spack/mistral-spack/etc/spack/mirrors.yaml
==> [2020-03-31-18:08:57.719289] Finding buildcaches in /mnt/lustre01/sw/sw-spack/spack-mirrors/build_cache
==> [2020-03-31-18:08:57.720449] No binary for libsigsegv found: installing from source
==> [2020-03-31-18:08:57.720608] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquiring]
==> [2020-03-31-18:08:57.721365] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquired]
==> [2020-03-31-18:08:58.338114] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Released]
==> [2020-03-31-18:08:58.672975] [TOOLCHAIN] MATCH Gcc
==> [2020-03-31-18:08:58.673248] [TOOLCHAIN] MATCH Nag
==> [2020-03-31-18:08:58.673392] [TOOLCHAINS] set(['Gcc', 'Nag'])
==> [2020-03-31-18:08:58.673608] Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]
==> [2020-03-31-18:08:58.723050] Error: TypeError: expected string or buffer

/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None


Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 814, in child_process
    setup_package(pkg, dirty=dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 700, in setup_package
    set_build_environment_variables(pkg, build_env, dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 365, in set_build_environment_variables
    implicit_rpaths = compiler.implicit_rpaths()
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 297, in implicit_rpaths
    link_dirs = self._get_compiler_link_paths(exe_paths)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 328, in _get_compiler_link_paths
    output=str, error=str))  # str for py2
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/util/executable.py", line 144, in __call__
    quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
  File "/usr/lib64/python2.6/re.py", line 142, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or buffer
==> [2020-03-31-18:08:58.723974] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquiring]
==> [2020-03-31-18:08:58.725271] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquired]
==> [2020-03-31-18:08:59.369817] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Released]
==> [2020-03-31-18:08:59.408534] ChildError: TypeError: expected string or buffer
/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None

Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 814, in child_process
    setup_package(pkg, dirty=dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 700, in setup_package
    set_build_environment_variables(pkg, build_env, dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 365, in set_build_environment_variables
    implicit_rpaths = compiler.implicit_rpaths()
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 297, in implicit_rpaths
    link_dirs = self._get_compiler_link_paths(exe_paths)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 328, in _get_compiler_link_paths
    output=str, error=str))  # str for py2
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/util/executable.py", line 144, in __call__
    quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
  File "/usr/lib64/python2.6/re.py", line 142, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or buffer

@adamjstewart
Copy link
Member

@becker33 was it you or @alalazo or @scheibelp who did the implicit rpaths stuff? I'm guessing this bug is specific to Python 2.6.

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 1, 2020

Any updates on this issue? Anyway to tackle it down?

@adamjstewart
Copy link
Member

Do you have access to a Python 3 installation you can use to run Spack with? You can do python3 $(which spack) to explicitly use Python 3 and see if the same issue occurs for other versions of Python.

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 2, 2020

I tried that with python3 run, but still the same error is popping up

lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None

Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 814, in child_process
    setup_package(pkg, dirty=dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 700, in setup_package
    set_build_environment_variables(pkg, build_env, dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 365, in set_build_environment_variables
    implicit_rpaths = compiler.implicit_rpaths()
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 297, in implicit_rpaths
    link_dirs = self._get_compiler_link_paths(exe_paths)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/compiler.py", line 328, in _get_compiler_link_paths
    output=str, error=str))  # str for py2
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/util/executable.py", line 144, in __call__
    quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/util/executable.py", line 144, in <listcomp>
    quoted_args = [arg for arg in args if re.search(r'^"|^\'|"$|\'$', arg)]
  File "/sw/rhel6-x64/python/python-3.5.2-gcc49/lib/python3.5/re.py", line 173, in search
    return _compile(pattern, flags).search(string)
TypeError: expected string or bytes-like object

@scheibelp
Copy link
Member

Hi @ajkotobi, would you be able to test if #15902 resolves this issue for you?

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 8, 2020

Hi @scheibelp, I gave it a try and got this

==> [2020-04-08-09:24:34.331246] [TOOLCHAINS] {'Gcc', 'Nag'}
==> [2020-04-08-09:24:34.331400] Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]
==> [2020-04-08-09:24:34.363249] A dependency has updated CPATH, this may lead pkg-config to assume that the package is part of the system includes and omit it when invoked with '--cflags'.
==> [2020-04-08-09:24:34.382373] Error: TypeError: 'NoneType' object is not iterable

/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None


Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 814, in child_process
    setup_package(pkg, dirty=dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 727, in setup_package
    for mod in pkg.compiler.modules:
TypeError: 'NoneType' object is not iterable
==> [2020-04-08-09:24:34.383120] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquiring]
==> [2020-04-08-09:24:34.384499] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquired]
==> [2020-04-08-09:24:34.719840] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Released]
==> [2020-04-08-09:24:34.764751] ChildError: TypeError: 'NoneType' object is not iterable
/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py:832, in child_process:
        829            tb_string = traceback.format_exc()
        830
        831            # build up some context from the offending package so we can
  >>    832            # show that, too.
        833            package_context = get_package_context(tb)
        834
        835            build_log = None

Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 814, in child_process
    setup_package(pkg, dirty=dirty)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 727, in setup_package
    for mod in pkg.compiler.modules:
TypeError: 'NoneType' object is not iterable

@scheibelp
Copy link
Member

scheibelp commented Apr 8, 2020

From looking at your configs I think I see a second Spack bug: the config doesn't include a setting for modules which our logic is tripping over (specifically, the config for the nag compiler), I'll update #15902 to account for that (no changes need to be made on your end).

@scheibelp
Copy link
Member

I have updated #15902 which may address your issue although I'm having trouble replicating it. I wanted to check a couple things:

Firstly, does spack --version give more-specific version information?

Secondly, is that config you posted the full config? I'm wondering if it's missing an entry, specifically for nag:

- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
      cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
      f77: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
      fc: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
    operating_system: rhel6
    target: x86_64
    flags:
      fflags: -mismatch
    spec: nag@7.0

When I create a similar config entry Spack complains that it is missing a specification for "modules". If that is what the file looks like, could you also include the config entry for nag that you get by running spack config blame compilers?

Thanks!

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 8, 2020

Hi @scheibelp,
spack --version is 0.13.3

and

spack config blame compilers

- compiler:
    environment: {}
    extra_rpaths: []
    flags:
      cflags: -O2 -g -march=native -mavx2
      fflags: -O2 -g -march=native -mavx2
      cxxflags: -O2 -g -march=native -mavx2
    modules: []
   operating_system: rhel6
   paths:
     cc: /sw/rhel6-x64/gcc/gcc-7.1.0/bin/gcc
     cxx: /sw/rhel6-x64/gcc/gcc-7.1.0/bin/g++
     f77: /sw/rhel6-x64/gcc/gcc-7.1.0/bin/gfortran
     fc: /sw/rhel6-x64/gcc/gcc-7.1.0/bin/gfortran
   spec: gcc@7.1.0
   target: x86_64
compiler:
   paths:
     cc: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gcc
     cxx: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/g++
     f77: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gfortran
     fc: /sw/spack-rhel6/gcc-7.4.0-thvkfz/bin/gfortran
   operating_system: rhel6
   target: x86_64
   modules: []
   environment: {}
   extra_rpaths: []
   flags:
     cflags: -O2 -g -march=native
     fflags: -O2 -g -march=native
     cxxflags: -O2 -g -march=native
   spec: gcc@7.4.0
compiler:
   paths:
     cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
     cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
     f77: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
     fc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
   operating_system: rhel6
   target: x86_64
   modules: []
   environment: {}
   extra_rpaths: []
   flags:
     cflags: -O2 -g -march=native
     fflags: -O2 -g -march=native
     cxxflags: -O2 -g -march=native
   spec: gcc@9.1.0
compiler:
   paths:
     cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
     cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
     f77: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
     fc: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
   operating_system: rhel6
   target: x86_64
   flags:
     fflags: -mismatch
   spec: nag@7.0

That's the whole compilers.yaml, for NAG c++ & cc using gcc-9.1.0

@scheibelp
Copy link
Member

spack --version is 0.13.3

Apologies: this Spack version is old enough that spack --version doesn't print out enough info for me. Can you include the latest git commit (e.g. the first commit message in git log)?

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 9, 2020

@scheibelp, after trying out the changes in #15902, I'm getting these

==> [2020-04-09-11:30:50.506536] Installing libsigsegv
==> [2020-04-09-11:30:50.506610] Searching for binary cache of libsigsegv
==> [2020-04-09-11:30:50.508240] Reading config file /mnt/lustre01/sw/sw-spack/mistral-spack/etc/spack/mirrors.yaml
==> [2020-04-09-11:30:50.527490] Finding buildcaches in /mnt/lustre01/sw/sw-spack/spack-mirrors/build_cache
==> [2020-04-09-11:30:50.528662] No binary for libsigsegv found: installing from source
==> [2020-04-09-11:30:50.528845] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquiring]
==> [2020-04-09-11:30:50.529800] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Acquired]
==> [2020-04-09-11:30:51.173858] READ LOCK: /sw/spack-rhel6/.spack-db/lock[0:0] [Released]
==> [2020-04-09-11:30:51.617686] [TOOLCHAIN] MATCH Gcc
==> [2020-04-09-11:30:51.617957] [TOOLCHAIN] MATCH Nag
==> [2020-04-09-11:30:51.618081] [TOOLCHAINS] set(['Gcc', 'Nag'])
==> [2020-04-09-11:30:51.618310] Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]
==> [2020-04-09-11:30:51.647426] A dependency has updated CPATH, this may lead pkg-config to assume that the package is part of the system includes and omit it when invoked with '--cflags'.
==> [2020-04-09-11:30:51.722893] Using cached archive: /mnt/lustre01/sw/sw-spack/var/spack/cache/_source-cache/archive/3a/3ae1af359eebaa4ffc5896a1aee3568c052c99879316a1ab57f8fe1789c390b6.tar.gz
==> [2020-04-09-11:30:51.728811] Staging archive: /mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/libsigsegv-2.12.tar.gz
==> [2020-04-09-11:30:51.731970] '/bin/tar' '-xf' '/mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/libsigsegv-2.12.tar.gz'
==> [2020-04-09-11:30:52.049382] Created stage in /mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo
==> [2020-04-09-11:30:52.051165] No patches needed for libsigsegv
==> [2020-04-09-11:30:52.051285] Building libsigsegv [AutotoolsPackage]
==> [2020-04-09-11:30:52.051492] WRITE LOCK: /mnt/lustre01/sw/sw-spack/spack-stage/k202129/.lock[2989457390885689522:1] [Acquiring]
==> [2020-04-09-11:30:52.052928] WRITE LOCK: /mnt/lustre01/sw/sw-spack/spack-stage/k202129/.lock[2989457390885689522:1] [Acquired]
==> [2020-04-09-11:30:52.053382] WRITE LOCK: /sw/spack-rhel6/.spack-db/prefix_lock[2769608663123352535:1] [Acquiring]
==> [2020-04-09-11:30:52.054150] WRITE LOCK: /sw/spack-rhel6/.spack-db/prefix_lock[2769608663123352535:1] [Acquired]
==> [2020-04-09-11:30:52.060837] Executing phase: 'autoreconf'
==> [2020-04-09-11:30:52.065187] Executing phase: 'configure'
==> [2020-04-09-11:30:55.354739] WRITE LOCK: /sw/spack-rhel6/.spack-db/prefix_lock[2769608663123352535:1] [Released]
==> [2020-04-09-11:30:55.355386] WRITE LOCK: /mnt/lustre01/sw/sw-spack/spack-stage/k202129/.lock[2989457390885689522:1] [Released]
==> [2020-04-09-11:30:55.371991] Error: ProcessError: Command exited with status 77:
    '/mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/spack-src/configure' '--prefix=/sw/spack-rhel6/libsigsegv-2.12-jtpubt' '--enable-shared'

2 errors found in build log:
     8     checking for a thread-safe mkdir -p... /bin/mkdir -p
     9     checking for gawk... gawk
     10    checking whether make sets $(MAKE)... yes
     11    checking whether make supports nested variables... yes
     12    checking for gcc... /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spac
           k/env/cc
     13    checking whether the C compiler works... no
  >> 14    configure: error: in `/mnt/lustre01/sw/sw-spack/spack-stage/k202129/
           spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/spack-s
           rc':
  >> 15    configure: error: C compiler cannot create executables
     16    See `config.log' for more details

See build log for details:
  /mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/spack-build-out.txt

Traceback (most recent call last):
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_environment.py", line 815, in child_process
    return_value = function()
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/package.py", line 1713, in build_process
    phase(self.spec, self.prefix)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/package.py", line 111, in phase_wrapper
    phase(spec, prefix)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/build_systems/autotools.py", line 277, in configure
    inspect.getmodule(self).configure(*options)
  File "/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/spack/util/executable.py", line 189, in __call__
    proc.returncode, long_msg)
ProcessError: Command exited with status 77:
    '/mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/spack-src/configure' '--prefix=/sw/spack-rhel6/libsigsegv-2.12-jtpubt' '--enable-shared'

@alalazo
Copy link
Member

alalazo commented Apr 9, 2020

@ajkotobi @scheibelp I'll reopen the issue here, as it seems not solved by #15902

@alalazo alalazo reopened this Apr 9, 2020
@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 9, 2020

As it's not from cloning git repo and I downloaded it as tar here

@scheibelp
Copy link
Member

  >> 15    configure: error: C compiler cannot create executables
     16    See `config.log' for more details

can you include any error messages you see from config.log?

@ajkotobi
Copy link
Contributor Author

ajkotobi commented Apr 9, 2020

Sure.

configure:3263: $? = 0
configure:3252: /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc -v >&5
Reading specs from /mnt/lustre01/sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/../lib/gcc/x86_64-pc-linux-gnu/9.1.0/specs
COLLECT_GCC=/sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
COLLECT_LTO_WRAPPER=/mnt/lustre01/sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/../libexec/gcc/x86_64-pc-linux-gnu/9.1.0/lto-wrapper
Target: x86_64-pc-linux-gnu
Configured with: /tmp/k202129/spack-stage/spack-stage-gcc-9.1.0-3m75hc27cu6z6f2sx7iqnvhhc5cdy2tc/spack-src/configure --prefix=/sw/spack-rhel6/gcc-9.1.0-3m75hc --with-pkgversion='Spack GCC' --with-bugurl=https://github.com/spack/spack/issues --disable-multilib --enable-languages=c,c++,fortran --disable-nls --with-mpfr=/sw/spack-rhel6/mpfr-3.1.6-6np5y7 --with-gmp=/sw/spack-rhel6/gmp-6.1.2-ivk6kr --with-system-zlib --with-sysroot=/ --with-stage1-ldflags='-Wl,-rpath,/sw/spack-rhel6/gcc-9.1.0-3m75hc/lib -Wl,-rpath,/sw/spack-rhel6/gcc-9.1.0-3m75hc/lib64 -Wl,-rpath,/sw/spack-rhel6/mpc-1.1.0-dxhcps/lib -Wl,-rpath,/sw/spack-rhel6/zlib-1.2.11-pogbcc/lib -Wl,-rpath,/sw/spack-rhel6/isl-0.20-jl4k5u/lib -Wl,-rpath,/sw/spack-rhel6/gmp-6.1.2-ivk6kr/lib -Wl,-rpath,/sw/spack-rhel6/binutils-2.32-ttki6p/lib -Wl,-rpath,/sw/spack-rhel6/mpfr-3.1.6-6np5y7/lib -static-libstdc++ -static-libgcc' --with-boot-ldflags='-Wl,-rpath,/sw/spack-rhel6/gcc-9.1.0-3m75hc/lib -Wl,-rpath,/sw/spack-rhel6/gcc-9.1.0-3m75hc/lib64 -Wl,-rpath,/sw/spack-rhel6/mpc-1.1.0-dxhcps/lib -Wl,-rpath,/sw/spack-rhel6/zlib-1.2.11-pogbcc/lib -Wl,-rpath,/sw/spack-rhel6/isl-0.20-jl4k5u/lib -Wl,-rpath,/sw/spack-rhel6/gmp-6.1.2-ivk6kr/lib -Wl,-rpath,/sw/spack-rhel6/binutils-2.32-ttki6p/lib -Wl,-rpath,/sw/spack-rhel6/mpfr-3.1.6-6np5y7/lib -static-libstdc++ -static-libgcc' --with-gnu-ld --with-ld=/sw/spack-rhel6/binutils-2.32-ttki6p/bin/ld --with-gnu-as --with-as=/sw/spack-rhel6/binutils-2.32-ttki6p/bin/as --with-mpc=/sw/spack-rhel6/mpc-1.1.0-dxhcps --with-isl=/sw/spack-rhel6/isl-0.20-jl4k5u
Thread model: posix
gcc version 9.1.0 (Spack GCC) 
configure:3263: $? = 0
configure:3252: /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc -V >&5
gcc: error: unrecognized command line option '-V'
gcc: fatal error: no input files
compilation terminated.
configure:3263: $? = 1
configure:3252: /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc -qversion >&5
gcc: error: unrecognized command line option '-qversion'; did you mean '--version'?
configure:3263: $? = 1
configure:3283: checking whether the C compiler works
configure:3305: /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc    conftest.c  >&5
/sw/spack-rhel6/binutils-2.32-ttki6p/bin/ld: unrecognized option '-Wl'
/sw/spack-rhel6/binutils-2.32-ttki6p/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status
configure:3309: $? = 1
configure:3347: result: no

configure: failed program was:
| /* confdefs.h */
| #define PACKAGE_NAME ""
| #define PACKAGE_TARNAME ""
| #define PACKAGE_VERSION ""
| #define PACKAGE_STRING ""
| #define PACKAGE_BUGREPORT ""
| #define PACKAGE_URL ""
| #define PACKAGE "libsigsegv"
| #define VERSION "2.12"
| /* end confdefs.h.  */
| 
| int
| main ()
| {
| 
|   ;
|   return 0;
| }
configure:3352: error: in `/mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-libsigsegv-2.12-jtpubt3it6326wujqjlpa35u7hzobxbo/spack-src':
configure:3354: error: C compiler cannot create executables
See `config.log' for more details

## ---------------- ##
## Cache variables. ##
## ---------------- ##

ac_cv_env_CC_set=set
ac_cv_env_CC_value=/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc
ac_cv_env_CFLAGS_set=
ac_cv_env_CFLAGS_value=
ac_cv_env_CPPFLAGS_set=
ac_cv_env_CPPFLAGS_value=
ac_cv_env_CPP_set=
ac_cv_env_CPP_value=
ac_cv_env_LDFLAGS_set=
ac_cv_env_LDFLAGS_value=
ac_cv_env_LIBS_set=
ac_cv_env_LIBS_value=
ac_cv_env_LT_SYS_LIBRARY_PATH_set=
ac_cv_env_LT_SYS_LIBRARY_PATH_value=
ac_cv_env_build_alias_set=
ac_cv_env_build_alias_value=
ac_cv_env_host_alias_set=
ac_cv_env_host_alias_value=
ac_cv_env_target_alias_set=
ac_cv_env_target_alias_value=
ac_cv_path_install='/usr/bin/install -c'
ac_cv_path_mkdir=/bin/mkdir
ac_cv_prog_AWK=gawk
ac_cv_prog_ac_ct_CC=/mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc
ac_cv_prog_make_make_set=yes
am_cv_make_support_nested_variables=yes

@scheibelp
Copy link
Member

I'm still looking at this - I hope to have an idea about the error message tomorrow.

@ajkotobi
Copy link
Contributor Author

Thanks

@scheibelp
Copy link
Member

(in short, I see the problem, but not necessarily a solution)

Regarding

configure:3283: checking whether the C compiler works
configure:3305: /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/cc    conftest.c  >&5
/sw/spack-rhel6/binutils-2.32-ttki6p/bin/ld: unrecognized option '-Wl'

This may be problematic now that I look into it: Spack's logic for nag attempts to account for its different linking mechanism - from Spack's nag.py:

    # Unlike other compilers, the NAG compiler passes options to GCC, which
    # then passes them to the linker. Therefore, we need to doubly wrap the
    # options with '-Wl,-Wl,,'

When the gcc/g++ executables are being used, this logic is wrong: ld ends up getting Wl directly, which confuses it. If you were to label this as a GCC compiler, like

compiler:
   paths:
     cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
     cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
     f77: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
     fc: /sw/spack-rhel6/nag-7.0-bjbr6o/bin/nagfor
   operating_system: rhel6
   target: x86_64
   flags:
     fflags: -mismatch
   spec: gcc@9.1-nag                  <---------(this is the only changed line)

then the opposite problem would occur when the nagfor executable was used.

@scheibelp
Copy link
Member

(Hopefully this question makes sense without reading the rest of this thread, although #15776 (comment) may provide some context)

@adamjstewart @davydden I noticed you two have done some work on the NAG compiler in Spack: do you have suggestions on using a hybrid NAG/GCC compiler to build packages in Spack?

@adamjstewart
Copy link
Member

@scheibelp @alalazo see #15474 where I started a lot of work to make the NAG compiler work. I'm still waiting for an answer to this question, which will solve the issue you're seeing here:

@alalazo it looks like you added the linker_arg logic in #9168. When does this arg get used? Maybe we don't even need a special linker_arg for NAG if it's only used by the linker, not the compiler.

It's the linker_arg that's the source of the -Wl, we need to get rid of. But I'm not sure if it's actually needed for NAG or not. Basically, right now, the NAG compiler doesn't work for any package because:

  1. We use this invalid linker_arg everywhere, and
  2. We use an invalid pic_flag everywhere

@scheibelp
Copy link
Member

#15474 was merged but the discussion implies there is still some work to be done before this issue is resolved. I will assign @alalazo since he reviewed that in part and also was discussed as responding to the follow-up logic with linker_arg (let me know if that doesn't work).

@scheibelp scheibelp removed their assignment Apr 27, 2020
@adamjstewart
Copy link
Member

@ajkotobi if you want to test my theory, open lib/spack/spack/compilers/nag.py and make the following change:

     @property
     def linker_args(self):
-        return '-Wl,-Wl,,'
+        return '-Wl,'

I think this should let you build OpenMPI. The only question is if this option is compiler-dependent or not. For example, it may be needed for Fortran, but it's definitely not needed for C/C++, which is why you encountered this error.

@ajkotobi
Copy link
Contributor Author

Hi @adamjstewart, I gave it a try and some dependencies got installed but on zlib got this

==> [2020-04-28-10:18:53.687349] Installing zlib
==> [2020-04-28-10:18:53.687463] Searching for binary cache of zlib
==> [2020-04-28-10:18:53.687815] Reading config file /.../etc/spack/mirrors.yaml
==> [2020-04-28-10:18:53.691366] Finding buildcaches in /.../spack-mirrors/build_cache
==> [2020-04-28-10:18:53.692087] No binary for zlib found: installing from source
==> [2020-04-28-10:18:54.522282] [TOOLCHAIN] MATCH Gcc
==> [2020-04-28-10:18:54.523151] [TOOLCHAIN] MATCH Nag
==> [2020-04-28-10:18:54.523256] [TOOLCHAINS] set(['Gcc', 'Nag'])
==> [2020-04-28-10:18:54.523456] Warning: microarchitecture specific optimizations are not supported yet on mixed compiler toolchains [check nag@7.0 for further details]
==> [2020-04-28-10:18:54.548990] A dependency has updated CPATH, this may lead pkg-config to assume that the package is part of the system includes and omit it when invoked with '--cflags'.
==> [2020-04-28-10:18:54.561404] [DEPRECATED METHOD]
"zlib" still defines the deprecated method "setup_environment" [should be split into "setup_build_environment" and "setup_run_environment"]
==> [2020-04-28-10:18:54.589644] Using cached archive: /.../var/spack/cache/_source-cache/archive/c3/c3e5e9fdd5004dcb542feda5ee4f0ff0744628baf8ed2dd5d66f8ca1197cb1a1.tar.gz
==> [2020-04-28-10:18:54.595614] Staging archive: /.../spack-stage/k202129/spack-stage-zlib-1.2.11-n6eucfkd2vrshsr5anovuup7b5wa6lhv/zlib-1.2.11.tar.gz
==> [2020-04-28-10:18:54.598258] '/bin/tar' '-xf' '/.../spack-stage/k202129/spack-stage-zlib-1.2.11-n6eucfkd2vrshsr5anovuup7b5wa6lhv/zlib-1.2.11.tar.gz'
==> [2020-04-28-10:18:55.124584] Created stage in /.../spack-stage/k202129/spack-stage-zlib-1.2.11-n6eucfkd2vrshsr5anovuup7b5wa6lhv
==> [2020-04-28-10:18:55.126472] No patches needed for zlib
==> [2020-04-28-10:18:55.126590] Building zlib [Package]
==> [2020-04-28-10:18:55.136685] Executing phase: 'install'
==> [2020-04-28-10:18:55.139094] './configure' '--prefix=/sw/spack-rhel6/zlib-1.2.11-n6eucf'
Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).
** ./configure aborting.
==> [2020-04-28-10:18:55.421268] WRITE LOCK: /sw/spack-rhel6/.spack-db/prefix_lock[4018513284803768721:1] [Released]
==> [2020-04-28-10:18:55.421924] WRITE LOCK: /mnt/lustre01/sw/sw-spack/spack-stage/k202129/.lock[1193679747334089225:1] [Released]
==> [2020-04-28-10:18:55.429138] Error: ProcessError: Command exited with status 1:
    './configure' '--prefix=/sw/spack-rhel6/zlib-1.2.11-n6eucf'
See build log for details:
  /.../spack-stage-zlib-1.2.11-n6eucfkd2vrshsr5anovuup7b5wa6lhv/spack-build-out.txt

Traceback (most recent call last):
  File "/.../lib/spack/spack/build_environment.py", line 815, in child_process
    return_value = function()
  File "/.../lib/spack/spack/package.py", line 1713, in build_process
    phase(self.spec, self.prefix)
  File "/.../lib/spack/spack/package.py", line 111, in phase_wrapper
    phase(spec, prefix)
  File "/.../var/spack/repos/builtin/packages/zlib/package.py", line 52, in install
  File "/.../lib/spack/spack/util/executable.py", line 189, in __call__
    proc.returncode, long_msg)
ProcessError: Command exited with status 1:
    './configure' '--prefix=/sw/spack-rhel6/zlib-1.2.11-n6eucf'

Is there anyway to workaround this?

./configure' '--prefix=/sw/spack-rhel6/zlib-1.2.11-n6eucf'
Compiler error reporting is too harsh for ./configure (perhaps remove -Werror).
** ./configure aborting.

Thanks

@alalazo alalazo added this to To do in Spack v0.14.3 release via automation Apr 28, 2020
@adamjstewart
Copy link
Member

@ajkotobi what version of Spack are you using? Can you try again on develop?

@becker33 becker33 added this to To Do in Spack v0.15.1 release via automation Jun 22, 2020
@becker33 becker33 removed this from To do in Spack v0.14.3 release Jun 22, 2020
@scheibelp scheibelp removed this from To Do in Spack v0.15.1 release Jul 8, 2020
@becker33 becker33 removed this from To Do in Spack v0.15.2 release Jul 23, 2020
@becker33 becker33 added this to To do in Spack v0.15.4 release via automation Jul 23, 2020
@tgamblin tgamblin added this to To do in Spack v0.15.5 release via automation Aug 13, 2020
@tgamblin tgamblin removed this from To do in Spack v0.15.4 release Aug 13, 2020
@ajkotobi
Copy link
Contributor Author

@adamjstewart @scheibelp , After moving to Spack v0.15.4 in our test system, and trying compilation with NAG, I've bump into following scenarios

When leave lib/spack/spack/compilers/nag.py as 'return -Wl,Wl,,' #15776 (comment), I faced

configure: error: C compiler cannot create executables

After changing it to

return '-Wl,'

compilation go through all dependencies successfully but during compilation, FORTRAN config section, it is facing

*** Fortran compiler
checking whether we are using the GNU Fortran compiler... no
checking whether /mnt/lustre01/sw/sw-spack/mistral-spack/lib/spack/env/nag/nagfor accepts -g... no
configure: WARNING: Open MPI now ignores the F77 and FFLAGS environment variables; only the FC and FCFLAGS environment variables are used.
checking whether ln -s works... yes
checking if Fortran compiler works... no
**********************************************************************
* It appears that your Fortran compiler is unable to produce working
* executables.  A simple test application failed to properly
* execute.  Note that this is likely not a problem with Open MPI,
* but a problem with the local compiler installation.  More
* information (including exactly what command was given to the
* compiler and what error resulted when the command was executed) is
* available in the config.log file in the Open MPI build directory.
**********************************************************************
configure: error: Could not run a simple Fortran program.  Aborting.
==> Error: ProcessError: Command exited with status 1:
    '/mnt/lustre01/sw/sw-spack/spack-stage/k202129/spack-stage-openmpi-3.1.6-zng42l5bqj3gkqpot5axjhqhzcz7uacz/spack-src/configure' '--prefix=/sw/spack-rhel6/openmpi-3.1.6-zng42l' '--enable-shared' '--disable-silent-rules' '--disable-oshmem' '--without-lustre' '--disable-builtin-atomics' '--enable-static' '--without-pmi' '--with-zlib=/sw/spack-rhel6/zlib-1.2.11-n6eucf' '--without-knem' '--without-hcoll' '--without-psm' '--without-ofi' '--without-cma' '--without-ucx' '--without-fca' '--without-mxm' '--without-verbs' '--without-xpmem' '--without-psm2' '--without-alps' '--without-lsf' '--without-sge' '--without-slurm' '--without-tm' '--without-loadleveler' '--disable-memchecker' '--with-hwloc=/sw/spack-rhel6/hwloc-1.11.11-s4rtkq' '--disable-java' '--disable-mpi-java' '--without-cuda' '--enable-wrapper-rpath' '--disable-wrapper-runpath' '--disable-mpi-cxx' '--disable-cxx-exceptions' '--with-wrapper-ldflags=-Wl,-rpath,/mnt/lustre01/sw/spack-rhel6/gcc-9.1.0-3m75hc/lib64'

1 error found in build log:
     369    * execute.  Note that this is likely not a problem with Open MPI,
     370    * but a problem with the local compiler installation.  More
     371    * information (including exactly what command was given to the
     372    * compiler and what error resulted when the command was executed) i
            s
     373    * available in the config.log file in the Open MPI build directory.
     374    *******************************************************************
            ***
  >> 375    configure: error: Could not run a simple Fortran program.  Aborting
            .

Recapping on compilers.yaml

compilers:
- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
      cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
      f77: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
      fc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gfortran
    operating_system: rhel6
    target: x86_64
    modules: []
    environment: {}
    extra_rpaths: []
    flags:
      cflags: -O2 -g -march=native
      fflags: -O2 -g -march=native
      cxxflags: -O2 -g -march=native
    spec: gcc@9.1.0
- compiler:
    paths:
      cc: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/gcc
      cxx: /sw/spack-rhel6/gcc-9.1.0-3m75hc/bin/g++
      f77: /sw/spack-rhel6/nag-7.0-p63pum/bin/nagfor
      fc: /sw/spack-rhel6/nag-7.0-p63pum/bin/nagfor
    operating_system: rhel6
    target: x86_64
    flags:
      fflags: -mismatch
    spec: nag@7.0
spack install -v openmpi%nag@7.0

Any ideas will be great.
Thanks

@adamjstewart
Copy link
Member

I think we're still waiting for a response from @alalazo on this one. It's unclear to me if linker_arg is used for both CC and FC. If it is used for both, we need to split it into cc_linker_arg and friends like we did for pic_flag and rpath_arg. Once we have better mixed compiler support, this will no longer be an issue, but I would definitely like to see this resolved before then, since it prevents NAG users from building literally anything with Spack.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

4 participants