-
Notifications
You must be signed in to change notification settings - Fork 708
/
trilinos.html
219 lines (190 loc) · 8.76 KB
/
trilinos.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN"
"http://www.w3.org/TR/html4/loose.dtd">
<html>
<head>
<title>The deal.II Readme on interfacing to Trilinos</title>
<link href="../screen.css" rel="StyleSheet">
<meta name="copyright" content="Copyright (C) 2008 - 2020 by the deal.II authors">
<meta name="keywords" content="deal.II">
</head>
<body>
<h1>Interfacing <acronym>deal.II</acronym> to Trilinos</h1>
<p>
<a href="https://trilinos.org/" target="_top">Trilinos</a> is a
software package that provides lots of functionality for linear
algebra, among other things. For example, it includes implementations of
a variety of linear solvers, as well as various different sparse and dense
matrix and vector formats. Trilinos also has many subpackages that deal
with problems that go far beyond linear algebra, for example nonlinear
solvers, automatic differentiation packages, uncertainty propagation
engines, etc. Of particular interest to deal.II is their ability to
provide this functionality both on sequential and parallel (using MPI)
computers.
Compared to <a href="http://www.mcs.anl.gov/petsc/" target="_top">PETSc</a>,
which is written in C, Trilinos is written in C++ and can be considered to
be a more modern version of PETSc though both packages are under
continuing development at their respective national laboratories.
</p>
<p>
<acronym>deal.II</acronym> has wrapper classes to the linear algebra
parts of Trilinos that provide almost the
same interfaces as the built-in <acronym>deal.II</acronym> linear
algebra classes. We use these interfaces for parallel computations based
on MPI since the native deal.II linear algebra classes lack this
ability. They are used, among other programs, in step-31 and step-32.
</p>
<p>
While building deal.II with Trilinos is covered in
the <a href="../readme.html">ReadMe file</a>, we here give an
introduction to building Trilinos in such a way that it contains
everything that we need from the <acronym>deal.II</acronym> side.
</p>
<h5>Installing Trilinos</h5>
<p style="color: red">
Note: The current version of deal.II requires at least Trilinos 12.4.
Deal.II is known to work with Trilinos up to 12.18. Other versions of
Trilinos should work too but have not been tested prior to the
release.
</p>
<p>
deal.II uses the following libraries from Trilinos:
<ul>
<li> Amesos,
<li> AztecOO,
<li> Epetra,
<li> EpetraExt (optional),
<li> Ifpack,
<li> ML,
<li> MueLu (optional, required for TrilinosWrappers::PreconditionAMGMueLu),
<li> ROL (optional),
<li> Sacado (optional),
<li> SEACAS (optional, required for GridIn::read_exodusii()),
<li> Teuchos,
<li> Tpetra (optional),
<li> Zoltan (optional).
</ul>
Trilinos uses <a href="http://cmake.org/">cmake</a> to configure and
build. The following slightly longish set of commands will set up a
reasonable configuration (<code>trilinos-x.y.z</code> is the
version-specific path into which the
Trilinos <code>.tar.gz</code> will unpack):
<pre>
cd trilinos-x.y.z
mkdir build
cd build
cmake \
-DTrilinos_ENABLE_Amesos=ON \
-DTrilinos_ENABLE_Epetra=ON \
-DTrilinos_ENABLE_EpetraExt=ON \
-DTrilinos_ENABLE_Ifpack=ON \
-DTrilinos_ENABLE_AztecOO=ON \
-DTrilinos_ENABLE_Sacado=ON \
-DTrilinos_ENABLE_SEACAS=ON \
-DTrilinos_ENABLE_Teuchos=ON \
-DTrilinos_ENABLE_MueLu=ON \
-DTrilinos_ENABLE_ML=ON \
-DTrilinos_ENABLE_ROL=ON \
-DTrilinos_ENABLE_Tpetra=ON \
-DTrilinos_ENABLE_COMPLEX_DOUBLE=ON \
-DTrilinos_ENABLE_COMPLEX_FLOAT=ON \
-DTrilinos_ENABLE_Zoltan=ON \
-DTrilinos_VERBOSE_CONFIGURE=OFF \
-DTPL_ENABLE_MPI=ON \
-DBUILD_SHARED_LIBS=ON \
-DCMAKE_VERBOSE_MAKEFILE=OFF \
-DCMAKE_BUILD_TYPE=RELEASE \
-DCMAKE_INSTALL_PREFIX:PATH=$HOME/share/trilinos \
../
make install
</pre>
You will need to adjust the path into which you want to install Trilinos
in the CMAKE_INSTALL_PREFIX line. If your computer has enough memory
available, it may also be useful to pass
the flag <code>-DTrilinos_ENABLE_EXPLICIT_INSTANTIATION=ON</code>, which
will improve compilation times of deal.II programs that use Trilinos.
</p>
<h5>Parallel builds</h5>
<p>
If your computer has more than one processor core, use
<code>make -jN</code> instead of <code>make</code> in the last line
above, where <code>N</code> is the number of processors you have.
</p>
<h5>BLAS and LAPACK</h5>
<p style="color: red">
Note: At the time of writing (November 2016) Trilinos (at least up to
v12.8.1) cannot link against LAPACK 3.6.0 or later, due to two symbols
that were deprecated (and removed by default) in LAPACK 3.6.0 (see the
<a href="http://www.netlib.org/lapack/lapack-3.6.0.html">
release notes</a>). To fix this, edit the Trilinos file
<code>packages/epetra/src/Epetra_LAPACK_wrappers.h</code> and change the
lines
<pre>
#define DGGSVD_F77 F77_BLAS_MANGLE(dggsvd,DGGSVD)
#define SGGSVD_F77 F77_BLAS_MANGLE(sggsvd,SGGSVD)
</pre>
to
<pre>
#define DGGSVD_F77 F77_BLAS_MANGLE(dggsvd3,DGGSVD3)
#define SGGSVD_F77 F77_BLAS_MANGLE(sggsvd3,SGGSVD3)
</pre>
before installing Trilinos. (Credit for this fix goes to
<a href="https://github.com/gahansen/Albany/wiki/ALCF-Vesta">this page</a>.)
</p>
<p>
Trilinos sometimes searches for other libraries but can't find
them if they are not in the usual directories or have other
names. A common example are BLAS or LAPACK. In a case like
this, you may have to specifically pass the directories and/or
library names under which they can be found
to <code>cmake</code>. For example, this may mean to add the
following flags to the call above:
<pre>
-DBLAS_LIBRARY_NAMES:STRING=goto \
-DBLAS_LIBRARY_DIRS:STRING=/apps/GotoBLAS/lib64 \
-DLAPACK_LIBRARY_NAMES:STRING=lapack \
-DLAPACK_LIBRARY_DIRS:STRING=/apps/lapack-3.2.1/lib64
</pre>
</p>
<h5>Using external direct solvers</h5>
<p>
Trilinos (via its Amesos package) can interface with a number of direct
solvers (see, for example,
<a href="http://trilinos.org/docs/r11.8/packages/amesos/doc/html/index.html"
target="_top">this page for Trilinos 11.8</a>). Most of them are external
packages to Trilinos and you will need to tell Trilinos configuration
scripts that you want to use them, for example via the
<code>TrilinosWrappers::SolverDirect</code> class. This can be tricky,
but adding defines similar to the following to the cmake command line
will achieve the goal to enable the UMFPACK and SuperLU/SuperLUDist
solvers:
<pre>
-DTPL_ENABLE_UMFPACK:BOOL=ON \
-DTPL_ENABLE_SuperLU:BOOL=ON \
-DTPL_ENABLE_SuperLUDist:BOOL=ON \
-DTPL_UMFPACK_INCLUDE_DIRS="/usr/include" \
-DSuperLUDist_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_DIST_3.2/SRC" \
-DTPL_SuperLUDist_LIBRARIES:FILEPATH="/path/to/SuperLU_DIST_3.2/lib/libsuperlu_dist.a" \
-DSuperLU_INCLUDE_DIRS:FILEPATH="/path/to/SuperLU_4.3/SRC" \
-DTPL_SuperLU_LIBRARIES:FILEPATH="/path/to/SuperLU_4.3/lib/libsuperlu_4.3.a"
</pre>
Similarly, to enable MUMPS, commands should include
<pre>
-DTPL_ENABLE_MUMPS:BOOL=ON \
-DTPL_ENABLE_SCALAPACK:BOOL=ON
</pre>
and possibly followed by
<pre>
-DTPL_MUMPS_INCLUDE_DIRS:PATH=/usr/include/openmpi-x86_64 \
-DSCALAPACK_LIBRARY_DIRS:PATH=/lib64/openmpi/lib \
</pre>
where you need to adjust the exact paths, of course.
</p>
<hr />
<div class="right">
<a href="http://validator.w3.org/check?uri=referer" target="_top">
<img style="border:0" src="http://www.w3.org/Icons/valid-html401" alt="Valid HTML 4.01!"></a>
<a href="http://jigsaw.w3.org/css-validator/check/referer" target="_top">
<img style="border:0;width:88px;height:31px" src="http://jigsaw.w3.org/css-validator/images/vcss" alt="Valid CSS!"></a>
</div>
</body>
</html>