Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spack cannot concretize R specs #2546

Closed
sknigh opened this issue Dec 9, 2016 · 14 comments
Closed

Spack cannot concretize R specs #2546

sknigh opened this issue Dec 9, 2016 · 14 comments

Comments

@sknigh
Copy link
Contributor

sknigh commented Dec 9, 2016

I need to build tk+X for as a dependency for R. I can't create a spec to do this that does not produce a CLI error.

$ spack spec R+X
Input spec
--------------------------------
R+X
...
Concretized
--------------------------------
==> Error: Invalid spec: 'cairo@1.14.0%gcc@4.8.5+X'. Package cairo requires variant ~X, but spec asked for +X

I also tried this, same result

$spack spec R ^tk+X
Input spec
--------------------------------
R
    ^tk+X
...
Concretized
--------------------------------
==> Error: Invalid spec: 'cairo@1.14.0%gcc@4.8.5+X arch=linux-rhel7-x86_64'. Package cairo requires variant ~X, but spec asked for +X

This too

$ spack spec R ^tk+X ^cairo~X
Input spec
--------------------------------
R
    ^cairo~X
    ^tk+X
...
Concretized
--------------------------------
==> Error: Invalid spec: 'tk@8.6.5%gcc@4.8.5+X arch=linux-rhel7-x86_64'. Package tk requires variant ~X, but spec asked for +X

Specing cairo+X AND cairo~X does not produce errors. I was able to spack install tk+X, cairo~X and cairo+X independently. I'm also able to produce errors like these on Ubuntu with d7e9134.

@citibeth
Copy link
Member

citibeth commented Dec 9, 2016

@hartzell This looks to me like a problem in the R package:

R/package.py:    depends_on('cairo~X', when='~X')
R/package.py:    depends_on('tk~X', when='~X')

Depending on things ~xyz is often problematic; why do you care that cairo was NOT built with X support? But that brings up...

Cairo (stylized as cairo) is a programming library that provides a vector graphics-based, device-independent API for software developers.

I don't believe that the Cairo API or Tk is tied into X. Therefore, shouldn't the variant on r be something like gui=tk?

@adamjstewart
Copy link
Member

This is the same problem I had in #1552 and solved in #1553. The problem is that R, tk, and cairo all default to +X, but pango defaults to ~X. @tgamblin This is another case where variant forwarding would save the day and make the dependencies way simpler.

@adamjstewart
Copy link
Member

For the time being, you have a few options. This works:

$ spack spec R ^pango+X

You can also set pango to default to +X in your packages.yaml. As for more permanent solutions that would help all users, you could set pango to default to +X. The only problem is that if a single user has any of these packages set to ~X in their packages.yaml, it doesn't work anymore. That was the case for me in #1552. For this reason, I opted to not forward the variants to each package. Instead, if you raise an error when trying to build +X with a ~X dependency, it should work how you would expect.

@sknigh
Copy link
Contributor Author

sknigh commented Dec 9, 2016

$ spack spec R ^pango+X

This works for me. Thanks!

@sknigh sknigh closed this as completed Dec 9, 2016
@hartzell
Copy link
Contributor

This quirk was discussed as part of the pull request that added X support to R, #2053.

The only way I could get my POC set of apps to build end-to-end last week (yay) was by doing what @adamjstewart suggested above (I actually spec'ed by cairo and pango because it worked when I tried it and then I moved on, I'll try @adamjstewart's shorter version).

But, I then needed to add similar prereq's for that R for each r-this and r-that package I installed.

ick(tm).

I have a half finished issue/question on my machine at work to see if I can do better while waiting for the improvements @adamjstewart mentions above.

@citibeth

I don't believe that the Cairo API or Tk is tied into X

Something needs to be cleaned up there too. I've always thought of Tk as an X windows toolkit, but apparently it can build on OS X w/out X. My current thought is that perhaps R would be build-able with/without Tk and that Tk should be window_system=X or some such.

I walked face-first into this topic because I needed to have R build with a Cairo that was built with X in order to get the PDF's that R generated to use glyphs for fonts instead of simply drawing bounding boxes (discovered while serving plots from an R shiny server) even though there isn't an X display anywhere in sight. I'm unclear why that should be so and suppose I'll have to invest some neurons into understanding the R build/assembly process.

I'm working towards a package for the open source R shiny server, which has prerequisites for node and npm (which is the topic of #2128) and me getting all of these bits to sing in harmony.

As always, thanks to @everyone (and the rest of you too!).

@citibeth
Copy link
Member

I looked up the arguments to R's configure (see below). It is my opinion that we would do well to go through that list carefully and be more careful about the command line arguments we pass to R's configure.

It is also my opinion that if you don't want to do X11 with R, then it should just depends_on('cairo'). When you don't specify a variant on a dependency, it means you don't care whether that variant is enabled or not; in this case, cairo could be satisfied by cairo+x or cairo~x. More specifically, I think package.py should look like:

    depends_on('cairo')
    depends_on('cairo+x', when='+x')

That would have avoided the problem in this Issue.

This will still lead to non-intuitive behavior: specifically, that if cairo defaults to +x, then you will still get an X11 build even if you just ask for r without requesting X11. If we're going to have a gotcha either way, I think this is the right choice. It should later be ameliorated/addressed by either:

  1. Go through packages and be consistent about whether they default +x or ~x.
  2. Make +x into a "univeral variant", as described in Re-work shared vs. static builds (Universal Variants) #2492. Then if you turn on +x once, you turn it on for all packages that use it.

I believe that (2) is really the right way to go; and could be done with zero or minimal changer to the current infrastructure. I'm beginning to believe it's a better alternative to variant forwarding (which seems messy).

`configure' configures R 3.3.2 to adapt to many kinds of systems.

Usage: ./configure [OPTION]... [VAR=VALUE]...

To assign environment variables (e.g., CC, CFLAGS...), specify them as
VAR=VALUE.  See below for descriptions of some of the useful variables.

Defaults for the options are specified in brackets.

Configuration:
  -h, --help              display this help and exit
      --help=short        display options specific to this package
      --help=recursive    display the short help of all the included packages
  -V, --version           display version information and exit
  -q, --quiet, --silent   do not print `checking ...' messages
      --cache-file=FILE   cache test results in FILE [disabled]
  -C, --config-cache      alias for `--cache-file=config.cache'
  -n, --no-create         do not create output files
      --srcdir=DIR        find the sources in DIR [configure dir or `..']

Installation directories:
  --prefix=PREFIX         install architecture-independent files in PREFIX
                          [/usr/local]
  --exec-prefix=EPREFIX   install architecture-dependent files in EPREFIX
                          [PREFIX]

By default, `make install' will install all the files in
`/usr/local/bin', `/usr/local/lib' etc.  You can specify
an installation prefix other than `/usr/local' using `--prefix',
for instance `--prefix=$HOME'.

For better control, use the options below.

Fine tuning of the installation directories:
  --bindir=DIR            user executables [EPREFIX/bin]
  --sbindir=DIR           system admin executables [EPREFIX/sbin]
  --libexecdir=DIR        program executables [EPREFIX/libexec]
  --sysconfdir=DIR        read-only single-machine data [PREFIX/etc]
  --sharedstatedir=DIR    modifiable architecture-independent data [PREFIX/com]
  --localstatedir=DIR     modifiable single-machine data [PREFIX/var]
  --libdir=DIR            object code libraries [EPREFIX/lib]
  --includedir=DIR        C header files [PREFIX/include]
  --oldincludedir=DIR     C header files for non-gcc [/usr/include]
  --datarootdir=DIR       read-only arch.-independent data root [PREFIX/share]
  --datadir=DIR           read-only architecture-independent data [DATAROOTDIR]
  --infodir=DIR           info documentation [DATAROOTDIR/info]
  --localedir=DIR         locale-dependent data [DATAROOTDIR/locale]
  --mandir=DIR            man documentation [DATAROOTDIR/man]
  --docdir=DIR            documentation root [DATAROOTDIR/doc/R]
  --htmldir=DIR           html documentation [DOCDIR]
  --dvidir=DIR            dvi documentation [DOCDIR]
  --pdfdir=DIR            pdf documentation [DOCDIR]
  --psdir=DIR             ps documentation [DOCDIR]

R installation directories:
  --libdir=DIR        R files to R_HOME=DIR/R [EPREFIX/$LIBnn]
    rdocdir=DIR       R doc files to DIR      [R_HOME/doc]
    rincludedir=DIR   R include files to DIR  [R_HOME/include]
    rsharedir=DIR     R share files to DIR    [R_HOME/share]

X features:
  --x-includes=DIR    X include files are in DIR
  --x-libraries=DIR   X library files are in DIR

System types:
  --build=BUILD     configure for building on BUILD [guessed]
  --host=HOST       cross-compile to build programs to run on HOST [BUILD]

Optional Features:
  --disable-option-checking  ignore unrecognized --enable/--with options
  --disable-FEATURE       do not include FEATURE (same as --enable-FEATURE=no)
  --enable-FEATURE[=ARG]  include FEATURE [ARG=yes]
  --enable-R-profiling    attempt to compile support for Rprof() [yes]
  --enable-memory-profiling
                          attempt to compile support for Rprofmem(),
                          tracemem() [no]
  --enable-R-framework[=DIR]
                          macOS only: build R framework (if possible), and
                          specify its installation prefix [no,
                          /Library/Frameworks]
  --enable-R-shlib        build the shared/dynamic library 'libR' [no]
  --enable-R-static-lib   build the static library 'libR.a' [no]
  --enable-BLAS-shlib     build BLAS into a shared/dynamic library [perhaps]
  --enable-maintainer-mode
                          enable make rules and dependencies not useful (and
                          maybe confusing) to the casual installer [no]
  --enable-strict-barrier provoke compile error on write barrier violation
                          [no]
  --enable-prebuilt-html  build static HTML help pages [no]
  --enable-lto            enable link-time optimization [no]
  --enable-java           enable Java [yes]
  --enable-byte-compiled-packages
                          byte-compile base and recommended packages [yes]
  --enable-static[=PKGS]  (libtool) build static libraries [default=no]
  --enable-shared[=PKGS]  (libtool) build shared libraries [default=yes]
  --enable-fast-install[=PKGS]
                          (libtool) optimize for fast installation
                          [default=yes]
  --disable-libtool-lock  avoid locking (might break parallel builds)
  --enable-long-double    use long double type [yes]
  --disable-openmp        do not use OpenMP
  --disable-largefile     omit support for large files
  --disable-nls           do not use Native Language Support

  --disable-rpath         do not hardcode runtime library paths

Optional Packages:
  --with-PACKAGE[=ARG]    use PACKAGE [ARG=yes]
  --without-PACKAGE       do not use PACKAGE (same as --with-PACKAGE=no)
  --with-blas             use system BLAS library (if available), or specify
                          it [no]
  --with-lapack           use system LAPACK library (if available), or specify
                          it [no]
  --with-readline         use readline library [yes]
  --with-aqua             macOS only: use Aqua (if available) [yes]
  --with-tcltk            use Tcl/Tk (if available), or specify its library
                          dir [yes]
  --with-tcl-config=TCL_CONFIG
                          specify location of tclConfig.sh []
  --with-tk-config=TK_CONFIG
                          specify location of tkConfig.sh []
  --with-cairo            use cairo (and pango) if available [yes]
  --with-libpng           use libpng library (if available) [yes]
  --with-jpeglib          use jpeglib library (if available) [yes]
  --with-libtiff          use libtiff library (if available) [yes]
  --with-system-tre       use system tre library (if available) [no]
  --with-valgrind-instrumentation
                          Level of additional instrumentation for Valgrind
                          (0/1/2) [0]
  --with-system-valgrind-headers
                          use system valgrind headers (if available) [no]
  --with-internal-tzcode  use internal time-zone code [no]
  --with-recommended-packages
                          use/install recommended R packages [yes]
  --with-ICU              use ICU library (if available) [yes]
  --with-pic[=PKGS]       (libtool) try to use only PIC/non-PIC objects
                          [default=use both]
  --with-aix-soname=aix|svr4|both
                          (libtool( shared library versioning (aka "SONAME")
                          variant to provide on AIX, [default=aix].
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-sysroot[=DIR]    Search for dependent libraries within DIR (or the
                          compiler's sysroot if not specified).
  --with-x                use the X Window System
  --with-gnu-ld           assume the C compiler uses GNU ld [default=no]
  --with-libpth-prefix[=DIR]  search for libpth in DIR/include and DIR/lib
  --without-libpth-prefix     don't search for libpth in includedir and libdir
  --with-included-gettext use the GNU gettext library included here [no]
  --with-libintl-prefix[=DIR]  search for libintl in DIR/include and DIR/lib
  --without-libintl-prefix     don't search for libintl in includedir and libdir

Some influential environment variables:
  R_PRINTCMD  command used to spool PostScript files to the printer
  R_PAPERSIZE paper size for the local (PostScript) printer
  R_BATCHSAVE set default behavior of R when ending a session
  MAIN_CFLAGS additional CFLAGS used when compiling the main binary
  SHLIB_CFLAGS
              additional CFLAGS used when building shared objects
  MAIN_FFLAGS additional FFLAGS used when compiling the main binary
  SHLIB_FFLAGS
              additional FFLAGS used when building shared objects
  MAIN_LD     command used to link the main binary
  MAIN_LDFLAGS
              flags which are necessary for loading a main program which will
              load shared objects (DLLs) at runtime
  CPICFLAGS   special flags for compiling C code to be turned into a shared
              object.
  FPICFLAGS   special flags for compiling Fortran code to be turned into a
              shared object.
  FCPICFLAGS  special flags for compiling Fortran 95 code to be turned into a
              shared object.
  SHLIB_LD    command for linking shared objects which contain object files
              from a C or Fortran compiler only
  SHLIB_LDFLAGS
              special flags used by SHLIB_LD
  DYLIB_LD    command for linking dynamic libraries which contain object files
              from a C or Fortran compiler only
  DYLIB_LDFLAGS
              special flags used for make a dynamic library
  CXXPICFLAGS special flags for compiling C++ code to be turned into a shared
              object
  SHLIB_CXXLD command for linking shared objects which contain object files
              from the C++ compiler
  SHLIB_CXXLDFLAGS
              special flags used by SHLIB_CXXLD
  SHLIB_FCLD  command for linking shared objects which contain object files
              from the Fortran 95 compiler
  SHLIB_FCLDFLAGS
              special flags used by SHLIB_FCLD
  TCLTK_LIBS  flags needed for linking against the Tcl and Tk libraries
  TCLTK_CPPFLAGS
              flags needed for finding the tcl.h and tk.h headers
  MAKE        make command
  TAR         tar command
  R_BROWSER   default browser
  R_PDFVIEWER default PDF viewer
  BLAS_LIBS   flags needed for linking against external BLAS libraries
  LAPACK_LIBS flags needed for linking against external LAPACK libraries
  LIBnn       'lib' or 'lib64' for dynamic libraries
  SAFE_FFLAGS Safe Fortran 77 compiler flags for e.g. dlamc.f
  r_arch      Use architecture-dependent subdirs with this name
  DEFS        C defines for use when compiling R
  JAVA_HOME   Path to the root of the Java environment
  R_SHELL     shell to be used for shell scripts, including 'R'
  YACC        The `Yet Another Compiler Compiler' implementation to use.
              Defaults to the first program found out of: `bison -y', `byacc',
              `yacc'.
  YFLAGS      The list of arguments that will be passed by default to $YACC.
              This script will default YFLAGS to the empty string to avoid a
              default value of `-d' given by some make applications.
  PKGCONF     path to pkg-config utility
  PKG_CONFIG_PATH
              directories to add to pkg-config's search path
  PKG_CONFIG_LIBDIR
              path overriding pkg-config's default search path
  CC          C compiler command
  CFLAGS      C compiler flags
  LDFLAGS     linker flags, e.g. -L<lib dir> if you have libraries in a
              nonstandard directory <lib dir>
  LIBS        libraries to pass to the linker, e.g. -l<library>
  CPPFLAGS    (Objective) C/C++ preprocessor flags, e.g. -I<include dir> if
              you have headers in a nonstandard directory <include dir>
  CPP         C preprocessor
  F77         Fortran 77 compiler command
  FFLAGS      Fortran 77 compiler flags
  CXX         C++ compiler command
  CXXFLAGS    C++ compiler flags
  CXXCPP      C++ preprocessor
  OBJC        Objective C compiler command
  OBJCFLAGS   Objective C compiler flags
  LT_SYS_LIBRARY_PATH
              User-defined run-time library search path.
  CXX1X       C++11 compiler command
  CXX1XSTD    special flag for compiling and for linking C++11 code, e.g.
              -std=c++11
  CXX1XFLAGS  C++11 compiler flags
  CXX1XPICFLAGS
              special flags for compiling C++11 code to be turned into a
              shared object
  SHLIB_CXX1XLD
              command for linking shared objects which contain object files
              from the C++11 compiler
  SHLIB_CXX1XLDFLAGS
              special flags used by SHLIB_CXX1XLD
  XMKMF       Path to xmkmf, Makefile generator for X Window System
  FC          Fortran compiler command
  FCFLAGS     Fortran compiler flags

Use these variables to override the choices made by `configure' or to help
it to find libraries and programs with nonstandard names/locations.

Report bugs to <https://bugs.r-project.org>.
R home page: <https://www.r-project.org>.

@citibeth citibeth reopened this Dec 12, 2016
@adamjstewart
Copy link
Member

I think package.py should look like:

depends_on('cairo')
depends_on('cairo+x', when='+x')

That would have avoided the problem in this Issue.

This doesn't fix anything. I tried commenting out the cairo~x dependency in R, cairo, and pango, but spack spec R still crashed.

  1. Make +x into a "univeral variant", as described in Re-work shared vs. static builds (Universal Variants) #2492. Then if you turn on +x once, you turn it on for all packages that use it.

I believe that (2) is really the right way to go; and could be done with zero or minimal changer to the current infrastructure. I'm beginning to believe it's a better alternative to variant forwarding (which seems messy).

I think we want the same thing here. Basically, if I ask for package +variant, that variant should be enabled for all dependencies as well. #2492 seems like kind of a hack to me. Would that mean that every single package in Spack would have a +X or +mpi variant that sometimes does something and sometimes doesn't? Would I have multiple installed copies of simple packages for +mpi and ~mpi even though they have no relation to mpi? It may be less code change, but that doesn't make it a better solution. Btw, I stopped following the shared vs. static vs. fpic discussion because there are like 10 different issues now that are just continuations of each other.

@citibeth
Copy link
Member

  1. Make +x into a "univeral variant", as described in Re-work shared vs. static builds (Universal Variants) #2492. Then if you turn on +x once, you turn it on for all packages that use it.

I believe that (2) is really the right way to go; and could be done with zero or minimal changer to the current infrastructure. I'm beginning to believe it's a better alternative to variant forwarding (which seems messy).

#2492 seems like kind of a hack to me.

There is a fine line between a hack and a parsimonious re-use of infrastructure. I'm not sure which one this is yet. :-)

Would that mean that every single package in Spack would have a +X or +mpi variant that sometimes does something and sometimes doesn't?

Let's see what it would mean by thinking through an example. For every universal variant we desire, we make a dummy package for it. So for the x universal variant, we might make a package called variant-x -- which will offer a SINGLE variant called +x (default False). So either you "install" variant-x+x or variant-x~x.

Now... packagaes that used to offer a variant +x will no longer do so. Instead of:

variant('x')

they will do:

depends_on('variant-x')

Similarly, when('+x') will change to when('^variant-x+x') and if 'x' in spec will change to if 'x' in spec['variant-x'].

The results of this would be:

  1. The +x variant is now injected from the bottom of the DAG, not the top. We are taking advantage of the fact that we already have a good way to "forward" variants up the DAG, but not down the DAG. Users would request mypackage^variant-x+x instead of mypackagke+x.

  2. On any one build, all packages that use +x now must have the same value of the variant. You can no longer link a+x with b~x.

With this in mind, answers to questions above:

Would that mean that every single package in Spack would have a +X or +mpi variant that sometimes does something and sometimes doesn't?

no

Would I have multiple installed copies of simple packages for +mpi and ~mpi even though they have no relation to mpi?

No. This is why each universal variant needs its own meta-package.

It may be less code change, but that doesn't make it a better solution.

I think the principles are sound, but the resulting syntax leaves a bit to be desired. Some Spack support on cleaning up the syntax would be helpful; syntactic suger is usually easier than deep reworkings of complex algorithms. But this could be as decent a way as any to implement universal variants / variant forwarding.

@hartzell
Copy link
Contributor

F.W.I.W. (and for future reference for me), here's how I've been building my R bits (just ran through that part of the build with commit e340f275).

# R things
spack install R@3.3.1+X^pango+X
spack install r-devtools@1.11.1^R+X ^pango+X
spack install r-shiny@0.13.2^R+X ^pango+X
spack install r-xml@3.98-1.5^R+X ^pango+X

@adamjstewart
Copy link
Member

@hartzell You can also put the following in ~/.spack/packages.yaml:

packages:
  R:
    variants: +X
  pango:
    variants: +X

and run:

$ spack install R
$ spack install r-devtools
$ spack install r-shiny
$ spack install r-xml

to get the same effect.

@hartzell
Copy link
Contributor

Yes, but that only works if the job runs as me, on a machine that has that file in my home directory.

The command I shared does not depend on any configuration outside of the repo. Making those changes in my home directory feels a bit like 'spooky magic at a distance', which usually leads to trouble....

I believe (have not tested) that I could get the same effect in an etc/spack/packages.yaml file in the repo, but that's only "permanent" if it's committed to the tree in which case I'll have to be working on a branch that tracks develop. I'll probably do something like that for internal releases, but so far have avoided it.

For now, being explicit works. Thanks for the follow up!

@ifelsefi
Copy link
Contributor

ifelsefi commented Jun 9, 2017

spack install r@3.4.0+X^pango+X %gcc@6.3.0

works!

I do not think X should be the default variant in headless HPC environment.

@becker33
Copy link
Member

becker33 commented Aug 7, 2017

Closing as no longer reproducible.

@becker33 becker33 closed this as completed Aug 7, 2017
@adamjstewart
Copy link
Member

It's no longer reproducible because I hacked every package to default to ~X. As soon as a single package defaults to +X, concretization grinds to a halt.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

6 participants