INSTALL

.. _TPLs:

=============================================
Amanzi Third Party Library (TPL) Build System
=============================================

Introduction
------------
This project builds the required and optional TPL software for the Amanzi code
project. The build system is designed to configure and build TPLs to the
requirements documented here https://software.lanl.gov/ascem/trac/wiki/Amanzi/Building/SoftwareRequirements.

Users are required to provide the following to build:

    * A recent and stable GCC or Intel compiler. We have successfully built with
      GNU versions 4.4, 4.5 and 4.6, and Intel versions 11 and 12. There
      are known issues with GCC 4.7.
    * Mercurial version 0.98 or higher.
    * CMake version 2.8.8 or higher.
    * MPI installation with compiler wrappers. (mpicc, mpicxx, etc.)
    * OpenSSL (required to build CURL) installation. This is available on
      most UNIX flavored systems and Macs.
    * BLAS/LAPACK built and tuned to the target architecture. See the 
      Trilinos Configuration section for more information.
    * A location to install all the TPLs. The entire software stack
      will be installed in directories rooted to this directory. Since the
      download--patch-build-install cycle for each TPL is tied to a single  
      target, the user must have read and write permission for this location.
    * The build directory will need approximately 2.1 Gb of space and the 
      install directory tree requires 300 Mb.

On most UNIX-like OS (including Mac OSX) systems, the entire TPL software stack can be built using the ``bootstrap.sh`` shell script found in the config directory. The script searches for CMake and will download and build CMake 2.8.8, if it does not locate the cmake command. The script also searches for a working mpi installation and will download and build OpenMPI 1.4.4 if it does not locate mpi.  At this time, the script does not build compilers, ``LAPACK`` or ``BLAS``. See the compilers_ section on how to obtain a supported compiler and LAPACK_ for ``LAPACK`` and ``BLAS`` suggestions. See the MPI_ section on how to obtain and install other mpi implementations.

The ``bootstrap.sh`` script will create two build directories, one for the TPLs, TPL_BUILD, and one for amanzi, amanzi-build, in the directory the script is called from. Users can control the install location of the TPLs and Amanzi through command line arguments.  The command ::

  <amanzi source root>/config/bootstrap.sh --help

will print all the available options and default values.

If a TPL fails to download, configure, build or install,  check the log files
stored under a directory labeled::
  
  <TPL name lowercase>/<TPL name lowercase>-timestamps.

Each step step is recorded with ``STDOUT`` and ``STDERR`` outputs stored
in separate files. Any bug report related to the TPLs should include these log files. 

.. _compilers:

Obtaining A Supported Compiler
++++++++++++++++++++++++++++++

At this time, Amanzi can be built with recent versions of GCC, 4.4 through 4.6, and
Intel versions 11.* and 12.*. Since GCC is widely used by the development team
and it is free, we recommend using one of the supported GCC compilers. 

GCC On Mac OSX
**************

The default GCC install on Mac OSX does not include a Fortran compiler. Mac users
should use the MacPorts software http://www.macports.org/ to install GCC. Once 
MacPorts is installed, ::

  port install gcc4x

will install GCC 4.x in the /opt/local directory. Be aware that Apple requires these
compiler names to be gcc-mp-4.x, g++-mp-4.x and gfortran-mp-4.x, not the standard names
gcc, g++ and gfortran.

GCC On Ubuntu
*************

Use the apt-get utility, ::

  apt-get install gcc

Building GCC From Source
************************

Please contact your local IT support before attempting to build GCC from source. The binaries may be
available in a non-standard location. 

Building GCC requires several precision libraries that must be installed before building GCC.
See http://gcc.gnu.org/install/prerequisites.html for more information. Once these
packages are installed, GCC must be configured with C, C++ and Fortran enabled. Documentation
of the available configure options is here http://gcc.gnu.org/install/configure.html. 
The quickest way to determine the best configure options is to find a similar 
machine with a matching OS that has the target GCC already installed, and look at
the verbose version output. The command ::

  gcc -v

will print out the configure options used to build that GCC compiler. One important note to 
remember: DO NOT BUILD GCC IN THE SOURCE DIRECTORY. Create a separate build directory.
The GCC build time is long. On most systems it will need more than 2 hours to complete.

.. _MPI:

Obtaining MPI
+++++++++++++

Amanzi has been built against both MPICH2 and OpenMPI. Both are easy to build and install,
but on clusters, users should contact their local support before building either one. Many
vendors provide tuned and optimized MPI libraries that should be used to maximize performance.
Otherwise, install either MPICH2 or OpenMPI.

MPICH2
******

We recommend that users install a stable version of MPICH2. MPICH even minor versions 
(Ex: 1.4.1 minor version is 4) are stable. Odd minor versions are beta releases.

Steps to build and install MPICH2 are

    1. Download MPICH2 http://www.mcs.anl.gov/research/projects/mpich2/downloads/index.php?s=downloads 

    2. Unpack the tar file and create a separate build directory. In this build directory,
       configure MPICH2 with::

         configure --prefix=<install prefix> \
                   --enable-f77 \
                   --enable-fc \
                   --enable-cxx \
                   --enable-romio \
                   --enable-shared \
                   --enable-static \
                   CC=<full path to C compiler>
                   CXX=<full path to C++ compiler>
                   FC=<full path to Fortran compiler>
                   F77=<full path to Fortran compiler that support F77>

    3. Build, test and install. MPICH does not support parallel builds.::

        make all; make test; make install

    4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system::

        setenv PATH <install prefix>/bin:${PATH}
        setenv LD_LIBRARY_PATH <install prefix>/lib:${LD_LIBRARY_PATH}

       or for ``bash`` shells::

          export PATH=<install prefix>/bin:${PATH}
          export LD_LIBRARY_PATH=<install prefix>/lib:${LD_LIBRARY_PATH}

       For  Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables::

        setenv PATH <install prefix>/bin:${PATH}
        setenv DYLD_LIBRARY_PATH <install prefix>/lib:${DYLD_LIBRARY_PATH}
    
       or for ``bash`` shells::

          export PATH=<install prefix>/bin:${PATH}
          export DYLD_LIBRARY_PATH=<install prefix>/lib:${DYLD_LIBRARY_PATH}

    5. Set environment variable MPI_ROOT. The ``bootstrap.sh`` script searches for compiler wrappers in this 
       directory when this is set.::

        setenv MPI_ROOT <install prefix> or export MPI_ROOT=<install prefix>

OpenMPI
*******
       
Steps to build and install OpenMPI are

    1. Download OpenMPI http://www.open-mpi.org

    2. Unpack the tar file in a directory and create a separate build directory. In this build directory
       configure OpenMPI with
       ::

         configure \
                   --prefix=<install prefix> \
                   --enable-mpi-f77 \
                   --enable-mpi-f90 \
                   --enable-mpi-cxx \
                   --enable-shared \
                   --enable-static \
                   CC=<full path to C compiler>
                   CXX=<full path to C++ compiler>
                   FC=<full path to Fortran compiler>
                   F77=<full path to Fortran compiler that support F77>,

       where the compilers are defined explicitly on the command line.           

    3. Build, test and install. OpenMPI does support parallel builds.::

        make all; make test; make install

    4. Set environment variables, PATH, LD_LIBRARY_PATH on UNIX/Linux system::

        setenv PATH <install prefix>/bin:${PATH}
        setenv LD_LIBRARY_PATH <install prefix>/lib:${LD_LIBRARY_PATH}

       or for ``bash`` shells::

          export PATH=<install prefix>/bin:${PATH}
          export LD_LIBRARY_PATH=<install prefix>/lib:${LD_LIBRARY_PATH}

       For  Mac OSX, set PATH and DYLD_LIBRARY_PATH environment variables::

        setenv PATH <install prefix>/bin:${PATH}
        setenv DYLD_LIBRARY_PATH <install prefix>/lib:${DYLD_LIBRARY_PATH}
    
       or for ``bash`` shells::

          export PATH=<install prefix>/bin:${PATH}
          export DYLD_LIBRARY_PATH=<install prefix>/lib:${DYLD_LIBRARY_PATH}

    5. Set environment variable MPI_ROOT. The bootstrap.sh script searches for compiler wrappers in this 
       directory when this is set. ::

        setenv MPI_ROOT <install prefix> or export MPI_ROOT=<install prefix>

.. _LAPACK:

Obtaining LAPACK/BLAS
+++++++++++++++++++++

Most systems have ``LAPACK`` and ``BLAS`` installed in system directories.
Building these libraries from source and optimizing for the target machine is difficult. 
We recommend that users attempt to build Amanzi before building ``LAPACK`` and ``BLAS`` from
source. Trilnos, the package that requires both, detects the libraries in their build system 
and in many cases finds them automatically. If the Trilinos build system fails to find the
libraries or if the user wants to build against non-standard ``LAPACK``/``BLAS`` distributions,
please read the Trilinos_ configuration section.

If the user must build ``LAPACK`` and ``BLAS`` from source, we recommend ``ATLAS``, a sofware package
that automatically builds tuned ``BLAS`` libraries and a small subset of ``LAPACK``. By default
``ATLAS`` does not build a full ``LAPACK`` distribution, but Amanzi requires a full ``LAPACK`` library.
Read the ``ATLAS`` documentation on how to build ``ATLAS`` with a full ``LAPACK`` installation.
``ATLAS`` downloads are available at http://math-atlas.sourceforge.net.


Building TPLs and Amanzi Using ``Boostrap.sh``
----------------------------------------------

The script ``bootstrap.sh`` is located in the ``config`` directory.  See ``bootstrap.sh --help`` for a full list of options and default settings.  The script will create two subdirectories in the currect directory.  The first is TPL_BUILD used for downloading and builing the TPLs.  The second is amanzi-build used for building Amanzi.  The TPLs and Amanzi will be installed in the directory specified by the option ``--prefix``.

See the :ref:`bootstrap` section for instructions building using ``Bootstrap.sh``.

Building TPLs From The Command Line using CMake
-----------------------------------------------

We recommend that users run the provided ``bootstrap.sh`` script in the ``config`` directory, to build
the TPLs. The instructions in this section are for advanced users and developers.

The steps to build entire suite of TPL software from the command line are

    1. Configure with cmake:: 

        cmake  \
               -D CMAKE_C_COMPILER=/full/path/mpicc \
	       -D CMAKE_CXX_COMPILER=/full/path/mpicxx \
	       -D CMAKE_Fortran_COMPILER=/full/path/mpif90 \
	       -D TPL_INSTALL_PREFIX=/full/path/to/tpls
	       /full/path/to/amanzi/config/SuperBuild

    2. Build with make (in parallel if desired)::
 
        make -j n

       where n is the number of make jobs.

    3. Install with make::

        make install

The final step, make install, copies a CMake configuration file,amanzi-tpl-config.cmake,
to the install location under share/cmake that is useful when building Amanzi. 
Passing this configuration to cmake using the -C option defines all the TPL locations
needed to build amanzi.

For example ::

 cmake -C /full/path/to/amanzi-tpl-config.cmake \ 
   [Additional CMake opts] /full/path/amanzi
 
will build and link amanzi linked against the TPLs built with this 
project, reducing the need for complicated build scripts.


Known Issues
------------


NetCDF fails to configure on Mac OSX
++++++++++++++++++++++++++++++++++++
This problem occurs when a user has HDF5 installed through MacPorts and uses
the MacPorts OpenMPI. This is an issue with NetCDF's configure script which
does not allow the user to control the HDF5 prefix installation. Users can either
install OpenMPI in another directory and use the wrappers in that directory (RECOMMENDED)
or set ENABLE_NetCDF4:BOOL=FALSE while configuring SuperBuild from the command line. 

TPL build fails with any PGI compiler
+++++++++++++++++++++++++++++++++++++
We have not found a PGI C++ compiler that builds our Boost configuration
successfully. TPL build system has successfully built with GCC 4.4, 4.5 and
4.6 and Intel 11.* and Intel 12.*.

Library parallelio (ASCEM-IO) missing archive index (MacOSX)
++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
This bug has only been reported on Macs. Users will experience this bug
when linking the HDF5 test binaries in the Amanzi test suite. Despite a kludge
in the shell script to address this when building ASCEM-IO, this problem persists.
The work-around is to run ranlib on the installed libparallelio.a library.

Fails to locate OpenSSL in the build CURL step
++++++++++++++++++++++++++++++++++++++++++++++
CURL needs the location of OpenSSL to enable downloads from secure sites.
There is a find_package call in the CURL build definition that attempts
to find the system OpenSSL. If this find fails, define the installation 
location of OpenSSL with ``-D OPENSSL_ROOT_DIR:FILEPATH=<directory>`` when 
configuring and rerun cmake.

.. _Trilinos:

Trilinos Configuration
----------------------

Amanzi relies on Trilinos for basic distributed data structures, input file
handling and non-linear solvers. It is critical, for performance reasons, that
the Trilinos build point to tuned BLAS and LAPACK libraries. On most systems,
the BLAS and LAPACK libraries are found in standard system libraries
(/usr/lib or /usr/lib64).  However on large scale clusters these libraries
are typically generic implementations and will not provide optimal performance.
Such machines have tuned BLAS/LAPACK libraries. Please consult the support
documentation for these machines to find the recommended libraries.

To guide the search for BLAS/LAPACK libraries, add ``-D ENABLE_BLA_Search:BOOL=TRUE``
to your cmake options. The CMake mechanism (FindBLAS and FindLAPACK) searches through
a list of known vendor supplied BLAS/LAPACK libraries.
Unless, ``BLA_VENDOR`` is set, all possible vendors
are searched, and the search ceases once a valid BLAS/LAPACK library is found. For the 
search to be successful, the library must be located in a standard installation location or
defined in the user's environment variables ``LD_LIBRARY_PATH`` or ``DYLD_LIBRARY_PATH``.

Supported BLAS/LAPACK Vendors:

============= ============================================
Vendor        Description
------------- --------------------------------------------
ATLAS         Automatically Tuned Linear Algebra Software
PhiPACK       PHiPAC (Portable High Performance ANSI C)
CXML          HP Libraries
DXML          ??
SunPerf       Sun Performance Libraries
SCSL          SGI/Cray Performance Libraries
SGIMATH       ??
IBMESSL       ??
ACML          AMD Core Math Library
ACML_MP       AMD Core Math Library multi-threaded
Apple         Mac OSX Accelerate Framework
NAS           ??
Intel10_32    Intel MKL with 32bit int
Intel10_64lp  Intel MKL with 64bit int
Intel         Older versions of MKL
Generic       Generic BLAS    
============= ============================================

If the search fails during the TPL configuration or if the Trilinos configuration
fails to locate these libraries, the user can override the search by explicitly
defining the libraries with ``BLAS_LIBRARIES`` and ``LAPACK_LIBRARIES`` at
configuration time. Multiple library names should be 
separated with a semi-colon. In some cases, Intel's MKL libraries, require
additional link flags when building Amanzi. Use the ``CMAKE_EXE_LINKER_FLAGS``
configuration variable to define these additional link flags when building Amanzi.
Consult the documentation for the particular library for more information. 


Trilinos Build Configuration Files
++++++++++++++++++++++++++++++++++

SuperBuild contains Trilinos CMake configuration files for specific platforms and
compilers. The files are located under the tpl-build/cmake directory.
The file name should be defined while configuring tpl-build with the option
``-D Trilinos_Build_Config_File:STRING=<file>``. This configuration file will 
be the last CMake configuration variable passed to Trilinos, thus all the 
definitions in this file will override previous configuration definitions.


:Hopper (NERSC):

                :Compiler: Intel
		:Config File: trilinos-build-libsci-intel.cmake
		:Description: Defines the LibSci libraries as BLAS/LAPACK 
                              libraries.


:Hopper (NERSC):

                :Compiler: GNU
		:Config File: trilinos-build-libsci-gnu.cmake
		:Description: Defines the LibSci libraries as BLAS/LAPACK 
                              libraries.