How to get spack to use system blas/lapack

[BarrySmith]

I tried to add

   blas:
        paths:
            blas: <system>
        buildable: False
    lapack:
        paths:
            lapack: <system>
        buildable: False

to my packages.py file as suggested by #571 but after adding it spack still goes off and tries to build openblas (which by the way is not particularly portable and often cannot be build which is why I want to use the system blas/lapack).

Is there a spack option that tells WHY spack is making each decision as to which package to build vs reuse etc?

[tgamblin]

blas is a virtual package so what should work is openblas: (or whatever the state blas is) instead. You can control which blas is used on the command line with ^netlib-blas, ^atlas, etc., or you can just add a providers section in packages.yaml (to tell it which blas to prefer). Look at etc/spack/defaults/packages.yaml To see an example -- that is where the default providers are set.

[tgamblin]

There isn't currently a good way to ask"why"... but I'd like to one day improve the concretizer to do that. spack config get packages should give you the merged packages.yaml. spack spec <spec> should give you details on what spack will install for a particular spec.

[tgamblin]

Oh and there is also spack graph which can help somewhat. Still not exactly "why".

[BarrySmith]

Looking at etc/spack/defaults/packages.yaml unfortunately doesn't help me understand how I should get it to use the system blas/lapack instead.

I tried putting

   all:
        providers:
            blas: [system]
            lapack: [system]

in packages.yaml and it still went off and tried to build the openblas!

When I run

$ ./spack config get packages
packages:
  all:
    providers:
      mpi:
      - openmpi
      - mpich
      blas:
      - system
      - openblas
      lapack:
      - system
      - openblas
      pil:
      - py-pillow
    paths: {}
    modules: &id001 {}
    buildable: true
    version: &id002 []
    compiler: &id003 []
  cmake:
    paths:
      cmake@2.8.12: /soft/apps/packages/cmake-2.8.12/bin/
    buildable: false
    providers: {}
    modules: *id001
    version: *id002
    compiler: *id003

and yet it ignores the system blas that it seems to know about.

[davydden]

to expand on @tgamblin answer, you can not specify virtual packages (blas, lapack, mpi and such) in packages.yaml. You need to do this for specific packages available in Spack, such as openblas, atlas, mkl, openmpi, mpich2, and so on. See http://spack.readthedocs.io/en/latest/configuration.html#external-packages for detailed explanation on how to do this.

As for the original question, there is no way to specify an abstract system blas/lapack, because there is no such package in Spack.

Keep in mind that setting up any external package which lives in system folders like /usr can lead to very ugly and hard to debug compiler errors because you then automatically expose /usr/bin and /usr/lib to configuration stage, as an example see #1224

[adamjstewart]

@BarrySmith I wrapped your code blocks so they would display indentation properly. Hope you don't mind.

[adamjstewart]

So it sounds to me like you'll either need to create a new Spack package for the blas/lapack you have on your system or choose an existing package that's close enough. Then, your packages.yaml will need to look like:

packages:
  all:
    providers:
      blas: [blas-package-name]
      lapack: [lapack-package-name]
  blas-package-name:
    paths: 
      blas-lapack-name@system: /path/to/blas
    version: [system]
    buildable: False
  lapack-package-name:
    paths: 
      lapack-lapack-name@system: /path/to/blas
    version: [system]
    buildable: False

In the above, replace blas-package-name and lapack-package-name with the name of the package you choose. These are usually the same package, but they don't have to be.

[tgamblin]

@adamjstewart: I think one of the existing providers would work and @BarrySmith shouldn't have to create a new package:

$ spack providers blas
    atlas                      mkl                          openblas
    intel-parallel-studio+mkl  netlib-lapack~external-blas
$ spack providers lapack
    atlas  intel-parallel-studio+mkl  mkl  netlib-lapack  openblas

We don't currently have a package specifically for apple BLAS, but wouldn't this work?

packages:
  all:
    providers:
      blas: netlib-lapack
      lapack: netlib-lapack
  netlib-lapack:
    paths: 
      netlib-lapack@system: /usr
    version: [system]
    buildable: False

If Apple BLAS uses really different library names, we might need to create a stub package for it, but I think you can fool Spack using the interface @alalazo and @davydden implemented for lapack & blas should handle this case (see #1682), assuming the libs are called libblas and liblapack.

@BarrySmith: FYI, for packages like blas and lapack that have an API spread out over potentially many obscurely named libraries that differ by implementation, we created an abstract interface that lets you query the DAG within Spack to get the library names. I think we need a name for that but I am not sure what we should call it. How does "polymorphic virtual dependencies" sound? Accurate, but too complicated? @alalazo @davydden?

Anyway, @BarrySmith: some explanation of the syntax: The all section is for defaults for all packages. So we're telling every package in Spack that it should prefer to use netlib-lapack as the provider for both lapack and blas. You could prefer different BLAS implementations for different packages like this:

packages:
  package1:
    providers:
      blas: netlib-lapack
      lapack: netlib-lapack
  package2:
    providers:
      blas: openblas
      lapack: openblas

The usage above is probably atypical for a single-user install, so you likely wan to stick with using all:, but at HPC centers we expect that certain codes are going to prefer certain MPI implementations, etc.

providers maps from virtual dependencies (e.g., mpi, lapack, bias, etc.) to real dependencies (e.g. openmpi, mpich, openblas, netlib-lapack). I dunno how finely you've combed the docs but basically when a package depends_on('blas'), that's a placeholder for something that provides('blas'). You can think of virtual dependencies as interfaces, where Spack handles swapping in a valid implementation of the interface. Or you can force one, e.g.:

spack install xsdk ^netlib-lapack
spack install xsdk ^openblas
spack install xsdk ^mkl

mkl might be a good one to look at, as it is a placeholder for a system mkl -- you have to specify that one in packages.yaml

[davydden]

@tgamblin good point, netlib-lapack has the default names for blas and lapack. So it should indeed work with system provided blas/lapack.

On mac-os there is indeed /usr/lib/libblas.dylib and /usr/lib/liblapack.dylib which point to /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib and /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib.

BUT: i would not use them as-is to avoid possible pollution of build environement with other libraries sitting in /usr/lib. Just manually create two symlinks in /opt/system_blas/lib and use

packages:
  all:
    providers:
      blas: netlib-lapack
      lapack: netlib-lapack
  netlib-lapack:
    paths: 
      netlib-lapack@system: /opt/system_blas
    version: [system]
    buildable: False

IMHO this is a much safer approach. Otherwise Cmake or alike may pick up some other system libraries instead of their Spack counterparts.

@citibeth : If we all agree that it is the way to do it, we may want to document this as a recommended way of using system's blas/lapack.

[tgamblin]

One last thing: @BarrySmith is having issues because he's trying to put virtual packages for the keys in packages.yaml sections. Spack should probably complain about that rather than silently doing nothing. I created #2069 for this.

[tgamblin]

BUT: i would not use them as-is to avoid possible pollution of build environement with other libraries sitting in /usr/lib. Just manually create two symlinks in /opt/system_blas/lib and use

This brings up another thing. I like the docs that @citibeth and @adamjstewart made because they lay bare a lot of Spack's rough edges and how to work around them. This is one of them that could be fixed easily.

Right now Spack will RPATH any system dependency you give it so @davydden's complaint is indeed a problem. But it doesn't have to.

If an external is in a system search path, we should just not RPATH it. You can find those on Linux by looking at /etc/ld.so.conf, and on Mac OS X the default search paths are embedded in dyld: /Library/Frameworks:/Network/Library/Frameworks:/System/Library/Frameworks and /usr/local/lib:/lib:/usr/lib. Not sure why /lib is preferred over /usr/lib, but whatever. We should just have the build environment exclude these from the automatic -I, -L, and -Wl,-rpath args. They'll still be found, but at a lower precedence than the ones supplied in args.

This will eliminate the need for stuff like the /false/path that @citibeth describes here. People can just write /usr.

[davydden]

Right now Spack will RPATH any system dependency you give it so @davydden's complaint is indeed a problem. But it doesn't have to.

👍 for this to be fixed and some unit tests added to check that /usr/lib is excluded from -I, -L, -Wl,-rpath and Cmake variables when a system package is used.

[tgamblin]

@davydden: I probably can't work on it today but i think it's not such a hard PR. You'd need to look at:

• lib/spack/spack/build_environment.py
• lib/spack/spack/test/cc
• lib/spack/env/cc

[citibeth]

I believe It's already documented that you should never put system paths inside packages.yaml. As others have noted, this can cause builds to pick up packages other than the ones Spack intended (i.e. it HAS caused me pain in the past). The following is problematic:

packages:
  all:
    providers:
      blas: netlib-lapack
      lapack: netlib-lapack
  netlib-lapack:
    paths: 
      netlib-lapack@system: /usr
    version: [system]
    buildable: False

Instead, change it to:

      netlib-lapack@system: /fake/path

This works because /usr is already searched by compilers.

@davydden outlines a more correct approach (also documented, I believe): Create a single-package tree made of symlinks into system stuff. Spack works best when all software is found in single-package trees.

[tgamblin]

@citibeth: see my comments above. That's a bug, but it's good that the workaround is documented.

[BarrySmith]

I can't for the life of me get this to work, both linux and mac. I've tried

packages:
    cmake:
        paths:
            cmake@2.8.12: /soft/apps/packages/cmake-2.8.12/bin/
        buildable: False
    python:
        paths:
            python@2.7.12: /usr/local/bin
        buildable: False
    boost:
        paths:
            boost@1.62.0: /usr/local
        buildable: False
    netlib-lapack:
        paths:
            netlib-lapack@system: /usr
        version: [system]
        buildable: False
    all:
        providers:
            mpi: [mpich]
            blas: [netlib-lapack]
            lapack: [netlib-lapack]

and many variants, with different directories, replacing system with 3.6.1 and it always fails with

$ ./spack install petsc
Traceback (most recent call last):
  File "./spack", line 202, in <module>
    main()
  File "./spack", line 179, in main
    return_val = command(parser, args)
  File "/Users/barrysmith/Src/spack/lib/spack/spack/cmd/install.py", line 103, in install
    specs = spack.cmd.parse_specs(args.package, concretize=True)
  File "/Users/barrysmith/Src/spack/lib/spack/spack/cmd/__init__.py", line 106, in parse_specs
    spec.concretize()  # implies normalize
  File "/Users/barrysmith/Src/spack/lib/spack/spack/spec.py", line 1329, in concretize
    self._expand_virtual_packages(),
  File "/Users/barrysmith/Src/spack/lib/spack/spack/spec.py", line 1246, in _expand_virtual_packages
    spec)
  File "/Users/barrysmith/Src/spack/lib/spack/spack/concretize.py", line 118, in choose_virtual_or_external
    candidates = self._valid_virtuals_and_externals(spec)
  File "/Users/barrysmith/Src/spack/lib/spack/spack/concretize.py", line 110, in _valid_virtuals_and_externals
    usable.sort(cmp=cmp_externals)
  File "/Users/barrysmith/Src/spack/lib/spack/spack/concretize.py", line 98, in cmp_externals
    return candidates.index(a) - candidates.index(b)
ValueError: netlib-lapack@system is not in list

Poking around with the debugger doesn't help, it sure looks like netlib-lapack is in candidates but I cannot determine why it fails. Could someone post an exact packages.yaml that works for them?

If I take out the

    netlib-lapack:
        paths:
            netlib-lapack@system: /usr
        version: [system]
        buildable: False

it does what I expect and downloads netlib-lapack instead of openblas.
All the other stuff in my packages.yaml does work, it uses the indicated CMake etc.

[tgamblin]

@mplegendre: is this related to the concretization issue #2071 you just fixed today?

@BarrySmith: does your version include that fix? If not I'll look at it and try to get it working.

[BarrySmith]

Yes it seems to resolve the problem. Thanks!

I am not closing the issue because I think you should document using system blas/lapack clearly in the ReadTheDocs

[davydden]

@BarrySmith : Keep in mind that using Apple's blas/lapack directly is not a good idea for packages which need single precision or complex arithmetic. That's why Homebrew-science use VecLibFort for many linear algebra packages. I am adding it to Spack #2080, but I have not tested it as I do not use it anymore.

[BarrySmith]

@davydden Thanks for the warning. Any good package, like PETSc :-) has work arounds that ./configure turns on for the Apple blas/lapack interface bugs, but I realize other packages might not.

I am actually using system blas/lapack on linux for testing things out.