Skip to content

9_BlasLapack

Jump to bottom Edit New page
Lori A. Burns edited this page Apr 28, 2017 · 8 revisions

STOP! This page is out-of-date and is superseded by the docs

BLAS and LAPACK libraries

In Psi4, BLAS and LAPACK control much of the speed and efficiency of the code since computational chemistry is essentially linear algebra on molecular systems.

Examples

  • #####Link Against Intel MKL on Linux

    source /theoryfs2/common/software/intel2015/bin/compilervars.csh intel64
setup

  • #####Link Against Auto-Detected on Mac

    setup

  • #####Link Against ACML on Linux

    setup --blas /opt/acml5.1.0/gfortran64_fma4_mp/lib/libacml_mp.a --lapack /opt/acml5.1.0/gfortran64_fma4_mp/lib/libacml_mp.a --extra-math-flags -lgfortran

Imposing your will upon CMake

  • #####Relevant setup Options:

    --mkl [{sequential,parallel,cluster}]
                      pass -mkl=STRING flag to the compiler and linker
                      [default: None] (default: none)
--blas [{auto,builtin,none,/full/path/lib.a}]
                      specify BLAS library; possible choices are "auto",
                      "builtin", "none", or full path [default: auto]
--lapack [{auto,builtin,none,/full/path/lib.a}]
                      specify LAPACK library; possible choices are "auto",
                      "builtin", "none", or full path [default: auto]
--extra-math-flags STRING
                      extra linker flags (usually -Llibdir -llibname type
                      arguments) [default: None]
--accelerate          build using Mac OS X Accelerate Framework [default:
                      False]

  • #####Relevant cmake Options:

    this is really complicated. worth adding?

Notes

  • Dependency Status: Required

  • ##### What are known contraindicated Intel MKL versions

    • Errors have been reported with 10 that are fixed by 11

The current math libraries detection system (written by Radovan Bast and Jonas Juselius) probes the system for BLAS/LAPACK implementations. The macros start searching based on the MATH_ROOT environment variable. On a system where Intel MKL is installed, this would point to the location of MKL:

# Defines various Intel-related environment variables (MKLROOT included)
source /opt/intel/bin/compilervars.sh intel64
export MATH_ROOT=$MKLROOT

Having set MATH_ROOT correctly should be enough, if you are OK with using MKL. I say should because it didn't work on my Mac, but it did on my Linux laptop... Psi4 doesn't have a builtin implementation of BLAS/LAPACK Instead of going all the way through the compilation to get a failure from the linker, I inserted a fatal error in the root CMakeLists.txt if no BLAS/LAPACK implementations are found. If you want to use another BLAS/LAPACK implementation, pass the flags -DBLAS_TYPE=name -DLAPACK_TYPE=name to the setup script. For example, the following works on Ubuntu 14.04 with these packages installed:

sudo apt-get install libblas3gf libblas-doc libblas-dev liblapack3gf liblapack-doc liblapack-dev libatlas-base-dev libatlas-dev libatlas-doc libatlas3-base
./setup -DBLAS_TYPE=ATLAS -DLAPACK_TYPE=ATLAS

This will link against ATLAS (might be useful for debug builds) Installing the documentation packages is not strictly necessary. If you do not install the ATLAS packages, you will anyway get a working (albeit possibly very slow) implementation of BLAS/LAPACK.

Using the --mkl option

It is possible to bypass math libraries detection by passing the --mkl flag to the setup script. The -mkl flag will be appended to the compiler flags. There is no need to define MATH_ROOT in this case. The --mkl option can take the values sequential or parallel or cluster. For an explanation of the difference read here

Using Mac OS X Accelerate Framework

I put an option to the setup script that forces usage of the Accelerate Framework for BLAS/LAPACK. Ideally, the math libraries detection macros should be modified to account for the presence of the framework. For the moment, the patch is enough:

./setup --accelerate

This option overrides any other math option given to the script.

Much of the speed and efficiency of the Psi4 program depends on the corresponding speed and efficiency of the available BLAS and LAPACK libraries (especially the former). In addition, the most common compilation problems involve these libraries. Users may therefore wish to consider the following BLAS and LAPACK recommendations when building Psi4:

  1. It is NOT wise to use the stock BLAS library provided with many Linux distributions like RedHat. This library is usually just the netlib distribution and is completely unoptimized. PSI4's performance will suffer if you choose this route.

    The choice of LAPACK is less critical, and so the unoptimized netlib distribution is acceptable. If you do choose to use the RedHat/Fedora stock BLAS and LAPACK, make sure that the blas-devel and lapack-devel packages are installed.

  2. Perhaps the best choice, if you have it available, is Intel's MKL library, which includes BLAS and LAPACK (note: use version 11 or later, we had reports of occasional errors using version 10). MKL is efficient and works well in threaded mode.

    Otherwise, the simplest choice is to use ATLAS (http://math-atlas.sourceforge.net/), which is readily available on all Linux distributions. Another alternative is OpenBLAS (https://github.com/xianyi/OpenBLAS, formerly GotoBLAS). These work well on nearly every architecture to which the Psi4 developers have access, though we have identified at least one case in which the Goto libraries yielded faulty DGEMM calls. On Mac OS X systems, the vecLib package that comes with Xcode works well.

    If you prefer to use the ACML (http://developer.amd.com/tools/cpu-development/amd-core-math-library-acml/) we highly recommend using the latest version. Older versions of ACML have been known to cause problems.

  3. Psi4 does not require a Fortran compiler, unless the resident BLAS and LAPACK libraries require Fortran-based system libraries. If you see compiler complaints about missing symbols like "do_fio" or "e_wsfe", then your libraries were most likely compiled with g77 or gfortran, which require -lg2c to resolve the Fortran I/O calls. Use of the same gcc package for Psi4 should normally resolve this problem.

  4. The Psi4 configure script can often identify and use several different BLAS and LAPACK libraries, but its ability to do this automatically depends on a number of factors, including correspondence between the compiler used for Psi4 and the compiler used to build BLAS/LAPACK, placement of the libraries in commonly searched directories, etc. Psi4's configure script will find your BLAS and LAPACK if any of the the following are installed in standard locations (e.g. /usr/local/lib):

    (a) ATLAS: libf77blas.a and libatlas.a, plus netlib's liblapack.a (b) MKL 8: libmkl.so and libmkl_lapack64.a (with the corresponding Intel compilers) (c) Goto: libgoto.a and netlib's liblapack.a (d) Cray SCSL (e.g. on SGI Altix): libscs.so (NB: No Fortran compiler is necessary in this case, so --with-fc=no should work.) (e) ESSL (e.g. on AIX systems): libessl.a

(5) If configure cannot identify your BLAS and LAPACK libraries automatically, you can specify them on the command-line using the --with-blas and --with-lapack arguments described above. Here are a few examples that work on the Psi4 developers' systems:

(a) Linux with ATLAS::

    --with-blas='-lf77blas -latlas' --with-lapack='-llapack -lcblas'

(b) Mac OS X with vecLib::

    --with-blas='-altivec -framework vecLib' --with-lapack=' '

(c) Linux with MKL 8.1 and icc/icpc/ifort 9.1::

    --with-libdirs=-L/usr/local/opt/intel/mkl/8.0.2/lib/32 --with-blas=-lmkl --with-lapack=-lmkl_lapack32

(d) Linux on ia32 with MKL 10.1 and icc/icpc 11.0::

    --with-blas='-Wl,--start-group -L/usr/local/opt/intel/mkl/10.1.0.015/lib/32 -l mkl -Wl,--end-group -lguide -lpthread'

  • Compilation notes for ATLAS

    These shortcut notes might be helpful if you are using Linux. However, we recommend reading and following the full ATLAS installation notes.

    You'll need a Fortran compiler installed.

    Unpack the source code, then make a compilation directory (could be an obj subdirectory in the source directory, or elsewhere).

    Turn off CPU throttling so the auto-tuning capabilities have a chance to work. On Linux, this can be tune using ::

    /usr/bin/cpufreq-selector -g performance

    cd into the compilation directory and run the source directory configure script there, with any necessary flags, e.g., ::

    /usr/local/src/atlas/configure --prefix=/usr/local/atlas

    where prefix gives the installation directory. It should automatically detect if you're on an x86_64

    Then make and check using ::

    make; make check; make ptcheck

    And install ::

    make install

  • Compilation notes for netlib's LAPACK

    These shortcut notes might be helpful if you are using Linux. However, we recommend reading and following the full LAPACK installation notes.

    You'll need a Fortran compiler installed.

    If you decide to compile LAPACK from source, it may be obtained from http://www.netlib.org/lapack/. Unpack the source code, and in the top-level source directory, you need to create a make.inc file with the appropriate options for your machine. For Linux/gfortran, simply ::

    cp make.inc.example make.inc

    Next, edit BLASLIB in make.inc to point to your BLAS library (full pathnames are recommended)::

    BLASLIB = /home/david/software/atlas3.9.25/lib/libf77blas.a /home/david/software/atlas3.9.25/lib/libatlas.a

    Edit Makefile as necessary (probably not needed). ::

    make

    Copy the resulting file [lapack_($ARCH).a] where you want it (a standard location like /usr/local/lib is easier for PSI to find). It is probably helpful to rename the file liblapack.a.

Add a custom footer
Add a custom sidebar
Clone this wiki locally