Installation
This chapter describes the manual installation of EDGE and its dependencies from scratch. If you are looking for an automated installation, consider skipping to sub-section Singularity Bootstrap or EDGE's support for cloud infrastructure in Sec. :doc:`../setup/cloud`.
Examples
EDGE's entire installation process is continuously tested. If you get stuck with this user guide, you might be able to find additional installation hints in the respective configurations:
- :edge_git:`Travis CI <blob/master/.travis.yml>`
- :edge_git:`GoCD <blob/master/tools/gocd/cruise-config.tmpl>`
- :edge_git:`Singularity <blob/master/tools/singularity/debian.def>`
General Remarks
EDGE links almost all libraries statically and expects corresponding library installations.
The descriptions below are adjusted accordingly and only building static versions is described.
This is one of the reasons why manual library installations are recommended, for example, not using sudo apt-get install
on your local machine.
libnuma
and compiler-provided libraries, e.g., OpenMP, are the only exception.
All of the instructions below assume that you initiate the installation of each library in EDGE's root-directory and will put the installed library into the directory libs
.
Make sure to navigate back to the root-directory before installing the next library.
Getting the Code
EDGE's sources are hosted at :edge_git:`GitHub <>`.
The repository has two branches :edge_git:`master <tree/master>` and :edge_git:`develop <tree/develop>`.
master
is the most recent stable version of EDGE.
The minmimum acceptance requirement for master
is passing EDGE's :edge_dev_pub:`continuous delivery pipeline </chapters/cont/cont.html>`.
Periodically EDGE also provides tags, which are simply snapshots of the master branch.
Typically tags passed additional, manual testing.
We recommend to use the most recent tag to get started with EDGE.
develop
is the bleeding edge version of EDGE.
Changes in develop
are intended to be merged into master.
However, develop
is for ongoing development and broken from time to time.
The procedure for obtaining the code as follows:
Clone the git-repository and navigate to the root-directory through:
git clone https://github.com/3343/edge.git cd edge
This gives you the
master
-branch. If this is what you want, jump over the next step.Checkout the desired tag:
git checkout YY.MM.VV
Remark:
YY.MM.VV
is a place holder. You have to replace this with your actual tag. Available tags are shown at the GitHub-homepage or directly throughgit tag
.Initialize and update the submodules:
git submodule init git submodule update
LIBXSMM
The single core backend of EDGE's high-performance kernels is provided through the library LIBXSMM. LIBXSMM is optional, but highly recommended due to severe performance-limitations of the vanilla kernels.
Install libxsmm by running:
cd submodules/libxsmm; PREFIX=../../libs make FORTRAN=0 BLAS=0 install
zlib
zlib is a requirement for the HDF5 library.
Download zlib from http://zlib.net/ (tar.gz):
wget http://zlib.net/zlib-1.2.11.tar.gz -O zlib.tar.gz
Extract zlib to the directory
zlib
:mkdir zlib; tar -xzf zlib.tar.gz -C zlib --strip-components=1
Configure the installation and set
libs
as installation directory by running:cd zlib; ./configure --static --prefix=$(pwd)/../libs
Run
make
to build the library andmake install
to put it in thelibs
directory.
HDF5
HDF5 is a requirement for point source descriptions. Futher, we recommend building MOAB, EDGE's interface to unstructured meshes, with HDF5-support. MOAB's native mesh format uses HDF5, which allows fast parsing of large meshes in parallel simulations.
Download HDF5 from https://www.hdfgroup.org/downloads/hdf5/source-code/ (gzip):
wget https://www.hdfgroup.org/package/gzip/?wpdmdl=13048 -O hdf5.tar.gz
Extract HDF5 to the directory
hdf5
:mkdir hdf5; tar -xzf hdf5.tar.gz -C hdf5 --strip-components=1
Configure the installation and set
libs
as installation directory.Sequential:
cd hdf5; ./configure --enable-shared=no --with-zlib=$(pwd)/../libs --prefix=$(pwd)/../libs
Parallel:
cd hdf5; ./configure --enable-shared=no --enable-parallel --with-zlib=$(pwd)/../libs --prefix=$(pwd)/../libs
Make sure to check that the configuration, printed at the very end, matches your expectations.
Finally run
make
to build the library andmake install
to put it in thelibs
directory.
MOAB
If using unstructured meshes in EDGE, you need to provide an installation of MOAB. Since ASCII-only builds of MOAB are troublesome, building with HDF5-support also for small-scale runs is recommended.
Generate the configure-script:
cd submodules/moab; autoreconf -fi
Configure the installation, two examples:
Sequential example using GNU compilers:
CXXFLAGS="-DEIGEN_DONT_VECTORIZE -fPIC" LIBS="$(pwd)/../../libs/lib/libz.a" CC=gcc CXX=g++ ./configure --disable-debug --disable-optimize --enable-shared=no --enable-static=yes --with-pic=yes --disable-fortran --enable-tools --disable-blaslapack --with-eigen3=$(pwd)/../eigen --with-zlib=$(pwd)/../../libs --with-hdf5=$(pwd)/../../libs --with-netcdf=no --with-pnetcdf=no --with-metis=yes --download-metis --prefix=$(pwd)/../../libs
MPI-parallel example using Intel compilers:
CXXFLAGS="-DEIGEN_DONT_VECTORIZE -fPIC" LIBS="$(pwd)/../../libs/lib/libz.a" CC=mpiicc CXX=mpiicpc ./configure --disable-debug --disable-optimize --enable-shared=no --enable-static=yes --with-pic=yes --disable-fortran --enable-tools --disable-blaslapack --with-eigen3=$(pwd)/../eigen --with-zlib=$(pwd)/../../libs --with-hdf5=$(pwd)/../../libs --with-netcdf=no --with-pnetcdf=no --with-metis=yes --download-metis --with-mpi --prefix=$(pwd)/../../libs
Now you can build MOAB with
make
and install it throughmake install
.
EDGE
EDGE uses SCons as build tool.
scons --help
returns all of EDGE's build-options.
All build options are given in the respective :ref:`sub-section <sec-setup-config-build>` of Sec. :doc:`../setup/config`.
You can enable the libraries in EDGE either by passing their installation directory explicitly (recommended) or by setting the environment variables CPLUS_INCLUDE_PATH
and LIBRARY_PATH
.
For example, let's assume that you installed LIBXSMM in the directory $(pwd)/libs
.
Than we could either enable LIBXSMM by passing xsmm=$(pwd)/libs
to EDGE's SCons-script or by using CPLUS_INCLUDE_PATH=$(pwd)/libs/include LIBRARY_PATH=$(pwd)/libs/lib scons [...] xsmm=yes
.
If something goes wrong with finding a library, EDGE will tell you so.
For example, if we did not install LIBXSMM in /tmp
, but tell EDGE so anyways, we get:
scons equations=elastic order=4 cfr=1 element_type=tet4 xsmm=/tmp
[...]
Checking for C++ static library libxsmmnoblas..no
Warning: Could not find libxsmm, continuing without.
Further information on what went wrong is logged in the file config.log
, which, in this case, shows that the compiler could not find the LIBXSMM-header:
[...] scons: Configure: Checking for C++ static library libxsmmnoblas.. .sconf_temp/conftest_2.cpp <- |#include <libxsmm.h> |int main(int i_argc, char **i_argv) { return 0; } g++ -o .sconf_temp/conftest_2.o -c -std=c++11 -Wall -Wextra -Wno-unknown-pragmas -Wno-unused-parameter -Werror -pedantic -Wshadow -Wundef -O2 -ftree-vectorize -DPP_N_CRUNS=1 -DPP_T_EQUATIONS_ELASTIC -DPP_T_ELEMENTS_TET4 -DPP_ORDER=4 -DPP_PRECISION=64 -I. -Isrc -I/tmp/include .sconf_temp/conftest_2.cpp .sconf_temp/conftest_2.cpp:1:21: fatal error: libxsmm.h: No such file or directory compilation terminated. scons: Configure: no
Stack Size
In certain settings EDGE allocates substantial amounts of data on the stack.
For high-order configurations, this memory is mostly occupied by thread-private global matrix structures.
To circumvent errors due to limited stacks on Linux systems use ulimit
. ulimit -s
shows you the current maximum, ulimit -s unlimited
allows unlimited sized stacks.
Server machines typically operate unlimited.
If running CentOS, you can obtain an unlimited stack as default by adding the following line to /etc/security/limits.conf
:
* - stack unlimited
Be aware, that unlimited
might interfere with your system's Out Of Memory (OOM) Killer.
Singularity Bootstrap
Singularity is software, which allows container-based execution of HPC-codes at close-to-native performance. EDGE provides a Debian-bootstrap for automated installation of different configurations:
Build Option | Enabled Bootstrap Cofingurations |
---|---|
element_type | tet4 (4-node tetrahedral elements) |
equations | elastic (elastic wave equations) |
order | 1 (FV), 2-5 (ADER-DG) |
cfr | 1 (hsw, non-fused), 16 (knl, fused) |
arch | hsw (Haswell), AVX512 (knl, skl, kml) |
xsmm | yes (LIBXSMM enabled except for FV) |
zlib | yes |
hdf5 | yes (enables kinematic sources) |
moab | yes (unstructured meshes), no (regular meshes) |
parallel | omp (shared memory parallelization) |
Once a container is generated, you can run it on systems with Singularity installed, without installing any further dependencies. Example systems with Singularity support are the XSEDE-resources Stampede and Comet. If you have root-access to a system with Singularity and debootstrap installed, you can generate a container containing EDGE and all its dependencies.
Adjust the EDGE-version in the bootstrap, if required (defaults to
develop
)Run the bootstrap to install the dependencies and EDGE-configurations:
sudo singularity build /tmp/edge.simg ./debian.def
The bootstrap might run for several hours, maybe grab a coffee.