release repository for ESPRESO
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
benchmarks
doc
env
include
src
tools/ASYNC
.gitignore
.gitlab-ci.yml
LICENCE
README.md
waf
wscript

README.md

ESPRESO

HIGHLY PARALLEL SOLVERS FOR ENGINEERING APPLICATIONS.

ESPRESO is an ExaScale PaRallel FETI SOlver developed at Czech national supercomputing centre IT4Innovations. Main focus of the development team is to create a highly efficient parallel solver which contains several FETI based algorithms including Hybrid Total FETI method suitable for parallel machines with tens or hundreds of thousands of cores. The solver is based on highly efficient communication layer on top of pure MPI.


Instalation


External Dependencies

In the current version the ESPRESO can be compiled and executed on a Linux operating system only. As of now, the library requires the Intel MKL and METIS libraries. These libraries have to be installed in the system before the ESPRESO installation process.

To compile and install the ESPRESO library a Python-based framework Waf is used. The compilation process has two phases: configuration and installation. The former configures persistent data and checks all required headers and libraries. The latter builds the library. If something in the environment is changed, it is mandatory to re-run the configuration process.

Building the library

The configuration process is controlled by a build.config script. The prefered way, how to create this script is to copy it from the install directory according to a preffered compiler. For compilation with e.g. Intel Compiler copy the appropriate script and the run configuration command:

$ cp install/build.config.icpc build.config
$ ./waf configure

After the configuration, the library can be installed:

$ ./waf install

This command builds all source files and creates the library. The type of the library can be controlled by the LIB_TYPE parameter in the configuration script.

Set up the environment

Before running the library, environment variables need to be set. The following variables should be set according to the number of CPU cores per compute node (nCores) and number of MPI processes processed per node (PPN):

  • OMP_NUM_THREADS - should be set to nCores/PPN
  • SOLVER_NUM_THREADS - should be set to nCores/PPN
  • PAR_NUM_THREADS - should be set to nCores/PPN
  • MKL_NUM_THREADS - in the current version it should be set to 1
  • CILK_NWORKERS - in the current version it should be set to 1

It is possible to set all environment variables at once usign the following script in the ESPRESO root directory:

$ . env/threading.default ${nCores/PPN}

Testing the installation

The installation of the library can be validated by the included set of benchmarks which can be executed as follows:

$ nosetest benchmarks

If all tests pass, the library is ready to use.


ESPRESO Solver Interface


The library contains a simple C API that allows to utilize FETI based solvers in third party softwares. The API has been successfully tested with an open source multiphysical simulation software Elmer FEM developed by CSC - IT Center for Science. The interface is available via the feti4i.h header in the include directory.

Call the library

Usage of the library is described by the simple example. The methods documentation can be found in the library header file.

#include "feti4i.h"

int main() {
    // Always initialize MPI before call ESPRESO!!
	MPI_Init(&argc, &argv);

	// Calling of provider should be replaced by appropriate methods in your library !!!
	espreso::APITestESPRESODataProvider provider(&argc, &argv);

	// Solving the problem by FETI4I works in 4 steps:
	//  1. Create the stiffness matrix
	//  2. Configure the ESPRESO library
	//  3. Create an instance of a problem
	//  4. Solve the instance


	// Step 1: Create the stiffness matrix.
	//       : Fill the empty matrix by element data
	FETI4IMatrix K;
	FETI4IInt    matrixType = provider.matrixType();
	FETI4IInt    indexBase = 0;

	FETI4ICreateStiffnessMatrix(&K, matrixType, indexBase); // create matrix
	for (size_t e = 0; e < provider.elements(); e++) {
		provider.addElementMatrix(K, e); // add element data
	}


	// Step 2: Configure the ESPRESO library
	//       : Set all options to default values
	//       : Change required parameters
	FETI4IInt  iopts[FETI4I_INTEGER_OPTIONS_SIZE];
	FETI4IReal ropts[FETI4I_REAL_OPTIONS_SIZE];

	FETI4ISetDefaultIntegerOptions(iopts); // set default integer options
	FETI4ISetDefaultRealOptions(ropts); // set default real options

	/* change the default values (see the list of available parameters below)
	iopts[FETI4IIntegerOptions::FETI4I_FETI_METHOD] = 1; // set HYBRID FETI
	iopts[FETI4IIntegerOptions::FETI4I_PRECONDITIONER] = 3; // set Dirichlet preconditioner
	*/


	// Step 3: Create an instance of a problem
	//       : Compute RHS
	//       : Fill L2G, Dirichlet, and neighbours
	FETI4IInstance            instance;
	std::vector<FETI4IReal>   rhs;
	std::vector<FETI4IInt>    dirichlet_indices;
	std::vector<FETI4IReal>   dirichlet_values;
	std::vector<FETI4IInt>    l2g;
	std::vector<FETI4IMPIInt> neighbours;

	provider.computeRHS(rhs);
	provider.fillL2G(l2g);
	provider.fillDirichlet(dirichlet_indices, dirichlet_values);
	provider.fillNeighbours(neighbours);

	FETI4ICreateInstance(
			&instance,
			K, // Stiffness matrix
			rhs.size(), rhs.data(), // RHS
			l2g.data(), // local 2 global indices mapping
			neighbours.size(), neighbours.data(), // neighbours clusters
			dirichlet_indices.size(), dirichlet_indices.data(), dirichlet_values.data(), // Dirichlet boundary condition
			iopts, ropts); // FETI4I options


	// Step 4: Solve the instance
	//       : Process the solution
	std::vector<FETI4IReal> solution(rhs.size());
	FETI4ISolve(instance, solution.size(), solution.data());


	// Finish: Destroy all data
	FETI4IDestroy(K);
	FETI4IDestroy(instance);

	MPI_Finalize();
}

List of available configuration parameters

FETI4IIntegerOptions::FETI4I_SUBDOMAINS = 4
    The number of subdomains per MPI process

FETI4IIntegerOptions::FETI4I_MAX_ITERATIONS = 200
    The maximal number of FETI Solver iterations

FETI4IIntegerOptions::FETI4I_FETI_METHOD = 0
    0 = TOTAL_FETI  -- The basis FETI method
    1 = HYBRID_FETI -- FETI with 2-level decomposition (suitable for huge examples)

FETI4IIntegerOptions::FETI4I_PRECONDITIONER = 3
    0 = NONE
    1 = LUMPED
    2 = WEIGHT_FUNCTION
    3 = DIRICHLET

FETI4IIntegerOptions::FETI4I_CGSOLVER = 0
    0 = PCG
    1 = PIPEPCG
    2 = ORTHOGONALPCG
    3 = GMRES
    4 = BICGSTAB

FETI4IIntegerOptions::FETI4I_N_MICS = 2
    The number of MIC accelerators per node

FETI4IIntegerOptions::FETI4I_VERBOSE_LEVEL = 1
    Verbosity of the library

FETI4IIntegerOptions::FETI4I_MEASURE_LEVEL = 0
    Level of time measurement

FETI4IIntegerOptions::FETI4I_PRINT_MATRICES = 0
    Turn on/off printing of assembler matrices

FETI4IRealOptions::FETI4I_PRECISION = 1e-5
    The requested solver precision

Set default parameters via a configuration file

The default parameters can be changed via a configuration file. The file have to be named espreso.ecf and should be placed at the run directory. The structure of the file corresponds with the parameters described above. The next example shows the configuration file with default parameters:

FETI4ILIBRARY {
  DOMAINS   4;

  SOLVER {
    #[TOTAL_FETI,HYBRID_FETI]
    METHOD          TOTAL_FETI;
    #[NONE,LUMPED,WEIGHT_FUNCTION,DIRICHLET]
    PRECONDITIONER   DIRICHLET;
    PRECISION            1E-05;
    MAX_ITERATIONS         200;
    #[PCG,PIPEPCG,ORTHOGONALPCG,GMRES,BICGSTAB]
    ITERATIVE_SOLVER       PCG;
    N_MICS                   2;
  }
}

Licence

See the LICENSE file at the root directory.