Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Parmetis (CMakePackage) doesn't always respect cflags #6858

Closed
KineticTheory opened this issue Jan 8, 2018 · 5 comments
Closed

Parmetis (CMakePackage) doesn't always respect cflags #6858

KineticTheory opened this issue Jan 8, 2018 · 5 comments
Labels

Comments

@KineticTheory
Copy link
Contributor

KineticTheory commented Jan 8, 2018

Summary

The current recipe for parmetis allows conflicting optimization levels to appear on the compile line in a way that isn't expected. For example: spack -d install parmetis%gcc cflags="-O3 -march=native" doesn't compile with the expected CFLAGS. I expect to see compile lines similar to this:

/usr/bin/gcc -O3 -march=native <include directives and defines> -o file.o -c file.c

Actual Result

/usr/bin/gcc -O3 -march=native <include directives and defines> -O2 -g -DNDEBUG -o file.o -c file.c

Because -O2 is listed after -O3 the optimization is not what I expect. I believe the -O2 -g -DNDEBUG flags are inserted by CMake as they are the default CMAKE_C_FLAGS_RELWITHDEBINFO values for gcc.

Possible solution

If SPACK_CFLAGS is set in the build environment, then zero-out the cmake defaulted CMAKE_C_FLAGS_XXX variables. #6839 demonstrates this model for metis and #6860 demonstrates this for parmetis, but I think it should be included in CMakePackage (possibly via the flag_handler function).

Related Issues

#6640
#6749
#6839
#6860

@KineticTheory KineticTheory changed the title Parmetis doesn't always respect cflags Parmetis (CMakePackage) doesn't always respect cflags Jan 8, 2018
KineticTheory added a commit to KineticTheory/spack that referenced this issue Jan 9, 2018
Background is provided in spack#6858. Essentially, this issue is that the command:

```
spack -d install parmetis%gcc` cflags="-O3 -march=native" \
   cxxflags="-O3 -march=native"
```

results in compile commands of the form:

`gcc -O3 -march=native <include directives> -O2 -g -o file.o -c file.c`

That is, the requested optimization level is ignored because CMake's initial
compiler flags for gcc are _appended_ to the compile line.  To avoid this
behavior, the initial `CMAKE_C_FLAGS` need to be disabled (set to an empty
string).

While I can do this locally for _parmetis_, it might make more sense to
implement a general fix for spack's `CMakePackage` class, possibly by modifying
the `flag_handler` functions (but I'm not sure how to implement this change).
@KineticTheory
Copy link
Contributor Author

As recommended in #6839, I added a flag_handler to the parmetis recipe:

@@ -53,6 +53,8 @@ class Parmetis(CMakePackage):
     # https://bitbucket.org/petsc/pkg-parmetis/commits/82409d68aa1d6cbc70740d0f35024aae17f7d5cb/raw/  # NOQA: E501
     patch('pkg-parmetis-82409d68aa1d6cbc70740d0f35024aae17f7d5cb.patch')
 
+    flag_handler = Package.build_system_flags
+
     def url_for_version(self, version):
         url = 'http://glaros.dtc.umn.edu/gkhome/fetch/sw/parmetis'
         if version < Version('3.2.0'):

But this doesn't work as expected:

% spack -d install parmetis cflags="-O0 -g" cxxflags="-O0 -g"

==> Installing parmetis
==> Error: TypeError: unbound method build_system_flags() must be called with Package instance as first argument (got str instance instead)
TypeError: TypeError: unbound method build_system_flags() must be called with Package instance as first argument (got str instance instead)

/scratch/kellyt/spack/lib/spack/spack/build_environment.py:683, in child_process:
     27    
     28                # build up some context from the offending package so we can
     29                # show that, too.
  >> 30                package_context = get_package_context(tb)
     31    
     32                build_log = None
     33                if hasattr(pkg, 'log_path'):


Traceback (most recent call last):
  File "/scratch/kellyt/spack/lib/spack/spack/build_environment.py", line 664, in child_process
    setup_package(pkg, dirty=dirty)
  File "/scratch/kellyt/spack/lib/spack/spack/build_environment.py", line 578, in setup_package
    set_compiler_environment_variables(pkg, spack_env)
  File "/scratch/kellyt/spack/lib/spack/spack/build_environment.py", line 168, in set_compiler_environment_variables
    injf, envf, bsf = pkg.flag_handler(flag, pkg.spec.compiler_flags[flag])
TypeError: unbound method build_system_flags() must be called with Package instance as first argument (got str instance instead)

It is very likely that my implementation is incorrect and I am looking for guidance to implement the flag_handler correctly.

cc: @becker33 @scheibelp

@scheibelp
Copy link
Member

Hi @KineticTheory we'll be looking at this later today or on 1/16. Your usage looks like what we'd intend so we need to look at whether there are any issues with the flag handler implementation.

@becker33
Copy link
Member

@scheibelp @KineticTheory there are two bugs here. One with Spack and one with the interaction between Spack and CMake. A fix for the first is up in #6960, the second will go up shortly.

@KineticTheory
Copy link
Contributor Author

@becker33 Thanks for diving into this issue. I'll watch the related PR and issues and close this issue when it is resolved.

@becker33
Copy link
Member

The other part of the fix is #6970. See issue #6961 for why it's a problem.

Once both are in, this should be resolved.

scheibelp pushed a commit that referenced this issue Jan 19, 2018
The flag_handlers method was being set as a bound method, but when
reset in the package.py file it was being set as an unbound method
(all python2 issues). This gets the underlying function information,
which is the same in either case.

The bug was uncovered for parmetis in #6858. This is a partial fix.
Included are changes to the parmetis package.py file to make use of
flag_handlers.
tgamblin pushed a commit that referenced this issue Jan 19, 2018
The flag_handlers method was being set as a bound method, but when
reset in the package.py file it was being set as an unbound method
(all python2 issues). This gets the underlying function information,
which is the same in either case.

The bug was uncovered for parmetis in #6858. This is a partial fix.
Included are changes to the parmetis package.py file to make use of
flag_handlers.
adamjstewart added a commit that referenced this issue Jan 9, 2020
* fixes #967

* Version bump to 0.9.1

- Bugfixes for spack find
- 0.9.1 can read specs from current develop.

* Don't assume spack is in the path when building docs.

* Quick fix for relocation issues.

* elf relocation fix: cherry-picked from develop branch (#6889)

* Revert "Quick fix for relocation issues."

This reverts commit 57608a6.

* Buildcache: relocate fixes (#6512)

* Updated function which checks if a binary file needs relocation.
  Previously this was incorrectly identifying ELF binaries as symbolic
  links (so they were being excluded from relocation). Added test to
  check that ELF binaries are not considered symlinks.

* relocate_text was not replacing paths in text files. Added test to
  check that text files are relocated properly (i.e. paths in the file
  are converted to the new prefix).

* Exclude backup files created by filter_file when installing from
  binary cache.

* Update write_buildinfo_file method signature to distinguish between
  the spec prefix and the working directory for the binary cache
  package.

* Final changes for v0.11.0 (#6318)

* Fix logo link in README.md to point to the develop branch. (#6969)

* Compiler flag handlers (#6415)

This adds the ability for packages to apply compiler flags in one of
three ways: by injecting them into the compiler wrapper calls (the
default in this PR and previously the only automated choice);
exporting environment variable definitions for variables with
corresponding names (e.g. CPPFLAGS=...); providing them as arguments
to the build system (e.g. configure).

When applying compiler flags using build system arguments, a package
must implement the 'flags_to_build_system_args" function. This is
provided for CMake and autotools packages, so for packages which
subclass those build systems, they need only update their flag
handler method specify which compiler flags should be specified as
arguments to the build system.

Convenience methods are provided to specify that all flags be applied
in one of the 3 available ways, so a custom implementation is only
required if more than one method of applying compiler flags is
needed.

This also removes redundant build system definitions from tutorial
examples

* Fix type issues with setting flag handlers (#6960)

The flag_handlers method was being set as a bound method, but when
reset in the package.py file it was being set as an unbound method
(all python2 issues). This gets the underlying function information,
which is the same in either case.

The bug was uncovered for parmetis in #6858. This is a partial fix.
Included are changes to the parmetis package.py file to make use of
flag_handlers.

* Bump version to 0.11.1

* Added flags to unit tests + OSX build done once per day (#6988)

* Adding flags to codecov reports

* OSX builds are triggered once a day

* Pull R list_urls from upstream.

* travis: removed /usr/local/include/c++ before installing gcc on OSX (#6515) (#7027)

"brew install gcc" fails for travis build because of an existing
/usr/local/include/c++. This commit removes the offending file
as suggested by brew.

* Fix gfortran 7 detection (#7017)

* Add NameError to exceptions caught from configure_args in module generation (#7173)

* Revert "Binary caching: remove symlinks, copy files instead (#9747)"

This reverts commit 058cf81.

* Make Spack relocate text files in build caches with relative binaries

* add the tfel package

* fix the tfel package

* fix the tfel package

* fix the tfel package

* Taking Adam J. Steward' remarks into account

* fixes trailing white spaces

* Update description

* Update dependencies following @adamjstewart adices

* Style fixes

* Style fixes

* Add java optional support

* add the maintainers attribute (following @alalazo advice), disable interface not selected (following @adamjstewart advice)

* flake8 fixes

* Fix Cast3M and python-bindings support. Python detection is made compatible with cmake'FindPythonLibs module (at least how it is used in TFEL)

* Style fixes

* Style fixes

* Fix test on python version

* Follow @adamjstewart advices: code is much cleaner and readable

* Small fix

* Small fix

* Add comment

* Small fix in cmake option

* try again (trying to overcome Travis CI unstable build process)

* Add support for the MFrontGenericInterfaceSupport project (MGIS)

* Style fixes

* Package documentation update

* Package documentation update

* Fix a typo thanks to Andreas Baumbach review

* Follow Adam J. Stewart advices

* Fix type

* bugfix: add back r's for invalid regexes

* tutorial basics section: fix gcc install version

* version bump: v0.12.1

* bugfix: bring in .travis.yml from develop

* Add new TFEL' versions (3.0.4, 3.1.4 and 3.2.1). Add new MGIS version (1.0.1). Fix MGIS dependency

* merge with spack:develop

* add missing dependency

* new versions of  and

* Fix MGIS url. Fix duplicate variant in TFEL

* Fix tfel packaging according to Adam J. Stewart' advices

* Fix flake8 warning

Co-authored-by: Massimiliano Culpo <massimiliano.culpo@gmail.com>
Co-authored-by: Todd Gamblin <tgamblin@llnl.gov>
Co-authored-by: Peter Scheibel <scheibel1@llnl.gov>
Co-authored-by: Greg Becker <becker33@llnl.gov>
Co-authored-by: Adam J. Stewart <ajstewart426@gmail.com>
JBlaschke pushed a commit to JBlaschke/spack that referenced this issue Jun 1, 2020
The flag_handlers method was being set as a bound method, but when
reset in the package.py file it was being set as an unbound method
(all python2 issues). This gets the underlying function information,
which is the same in either case.

The bug was uncovered for parmetis in spack#6858. This is a partial fix.
Included are changes to the parmetis package.py file to make use of
flag_handlers.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

No branches or pull requests

4 participants