-
Notifications
You must be signed in to change notification settings - Fork 471
/
INSTALL
1132 lines (910 loc) · 46.1 KB
/
INSTALL
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
377
378
379
380
381
382
383
384
385
386
387
388
389
390
391
392
393
394
395
396
397
398
399
400
401
402
403
404
405
406
407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
453
454
455
456
457
458
459
460
461
462
463
464
465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
592
593
594
595
596
597
598
599
600
601
602
603
604
605
606
607
608
609
610
611
612
613
614
615
616
617
618
619
620
621
622
623
624
625
626
627
628
629
630
631
632
633
634
635
636
637
638
639
640
641
642
643
644
645
646
647
648
649
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664
665
666
667
668
669
670
671
672
673
674
675
676
677
678
679
680
681
682
683
684
685
686
687
688
689
690
691
692
693
694
695
696
697
698
699
700
701
702
703
704
705
706
707
708
709
710
711
712
713
714
715
716
717
718
719
720
721
722
723
724
725
726
727
728
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
758
759
760
761
762
763
764
765
766
767
768
769
770
771
772
773
774
775
776
777
778
779
780
781
782
783
784
785
786
787
788
789
790
791
792
793
794
795
796
797
798
799
800
801
802
803
804
805
806
807
808
809
810
811
812
813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
925
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
969
970
971
972
973
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
Finite Element Discretization Library
__
_ __ ___ / _| ___ _ __ ___
| '_ ` _ \ | |_ / _ \| '_ ` _ \
| | | | | || _|| __/| | | | | |
|_| |_| |_||_| \___||_| |_| |_|
https://mfem.org
This file provides a detailed description of how to build and install the MFEM
library. For a simple build, see the step-by-step instructions on the website
at https://mfem.org/building.
The MFEM library has a serial and an MPI-based parallel version, which largely
share the same code base. The only prerequisite for building the serial version
of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
requires an MPI C++ compiler, as well as the following external libraries:
- hypre (a library of high-performance preconditioners)
https://github.com/hypre-space/hypre
- METIS (a family of multilevel partitioning algorithms)
https://github.com/mfem/tpls
Note: We recommend our mirror of metis-4.0.3/5.1.0 above because the METIS
webpage, http://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down
and we don't support yet the new repo https://github.com/KarypisLab/METIS.
The hypre dependency can be downloaded as a tarball from GitHub or from the
project webpage https://www.llnl.gov/casc/hypre. For example, the 2.24.0 release
of hypre is available at
https://github.com/hypre-space/hypre/archive/v2.24.0.tar.gz
The METIS dependency can be disabled but that is not generally recommended, see
the option MFEM_USE_METIS.
MFEM also includes support for devices such as GPUs, and programming models such
as CUDA, HIP, OCCA, OpenMP and RAJA.
- Starting with version 4.0, MFEM requires a C++11 compiler. We recommend using
a newer compiler, e.g. GCC version 4.9 or higher.
- CUDA support requires an NVIDIA GPU and an installation of the CUDA Toolkit
https://developer.nvidia.com/cuda-toolkit
- HIP support requires an AMD GPU and an installation of the ROCm software stack
https://rocmdocs.amd.com
- OCCA support requires the OCCA library
https://libocca.org
- OpenMP support requires a compiler implementing the OpenMP API
https://www.openmp.org
- RAJA support requires installation of the RAJA performance portability layer
with (optionally) support for CUDA and OpenMP
https://github.com/LLNL/RAJA
The library supports two build systems: one based on GNU make, and a second one
based on CMake. Both build systems are described below. Some hints for building
without GNU make or CMake can be found at the end of this file.
In addition to the native build systems, MFEM packages are also available in the
following package managers:
- Spack, https://github.com/spack/spack
- OpenHPC, http://openhpc.community
- Conda-forge, https://conda-forge.org (pre-built binaries linked with
OpenMPI/MPICH, hypre, and METIS)
We also recommend downloading and building the MFEM-based GLVis visualization
tool which can be used to visualize the meshes and solution in MFEM's examples
and miniapps. See https://glvis.org and https://mfem.org/building.
Quick start with GNU make
=========================
See also: https://mfem.org/building
Serial build:
make serial -j 4
Parallel build:
(download hypre and METIS 4 from above URLs)
(build METIS 4 in ../metis-4.0 relative to mfem/)
(build hypre in ../hypre relative to mfem/)
make parallel -j 4
(For METIS 5, see https://mfem.org/building/#parallel-build-using-metis-5)
CUDA build:
make cuda -j 4
(build for a specific compute capability: 'make cuda -j 4 CUDA_ARCH=sm_70')
HIP build:
make hip -j 4
(build for a specific AMD GPU chip: 'make hip -j 4 HIP_ARCH=gfx900')
Example codes (serial/parallel, depending on the build):
cd examples
make -j 4
Build everything (library, examples and miniapps) with current configuration:
make all -j 4
Quick-check the build by running Example 1/1p (optional):
make check
Quick start with CMake
======================
Serial build:
mkdir <mfem-build-dir> ; cd <mfem-build-dir>
cmake <mfem-source-dir>
make -j 4 (assuming "UNIX Makefiles" generator)
Parallel build:
(download hypre and METIS 4 from above URLs)
(build METIS 4 in ../metis-4.0 relative to mfem/)
(build hypre in ../hypre relative to mfem/)
mkdir <mfem-build-dir> ; cd <mfem-build-dir>
cmake <mfem-source-dir> -DMFEM_USE_MPI=YES
make -j 4
(For METIS 5, see https://mfem.org/building/#parallel-build-using-metis-5)
CUDA build:
(this build requires CMake 3.8 or newer)
mkdir <mfem-build-dir> ; cd <mfem-build-dir>
cmake <mfem-source-dir> -DMFEM_USE_CUDA=YES
make -j 4
Example codes (serial/parallel, depending on the build):
make examples -j 4
Build everything (library, examples and miniapps) with current configuration:
make exec -j 4
Quick-check the build by running Example 1/1p (optional):
make check
Building with GNU make
======================
The MFEM build system consists of two steps: configuration and compilation.
The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to "./configure". It is performed by running
make config [OPTIONS] ...
The OPTIONS are of the form VARIABLE=VALUE. Detailed description of the
configuration options is given below. Alternatively, the options can be
specified with an input file:
cp config/defaults.mk config/user.mk
(edit config/user.mk)
make config
Note that config/user.mk, if present, is loaded after config/defaults.mk and
its path/name can be changed with
make config USER_CONFIG=<user_config_file>
The build system can be configured to use a separate build directory, for an
out-of-source build. There are two ways to do that: the first one is
mkdir <mfem-build-dir> ; cd <mfem-build-dir>
make -f <mfem-source-dir>/makefile config [OPTIONS] ...
The second one is
cd <mfem-source-dir>
make BUILD_DIR=<mfem-build-dir> config [OPTIONS] ...
Note that in both cases the default location for the (optional) user
configuration file is <mfem-build-dir>/config/user.mk.
Once configured, the library can be built simply with
cd <mfem-build-dir> (if building out-of-source)
make
Note that re-configuration is only needed to change the currently configured
options. Several shortcut targets combining (re-)configuration and compilation
are also defined:
make serial -> Builds serial optimized version of the library
make parallel -> Builds parallel optimized version of the library
make debug -> Builds serial debug version of the library
make pdebug -> Builds parallel debug version of the library
make cuda -> Builds serial cuda optimized version of the library
make pcuda -> Builds parallel cuda optimized version of the library
make cudebug -> Builds serial cuda debug version of the library
make pcudebug -> Builds parallel cuda debug version of the library
make hip -> Builds serial hip optimized version of the library
make phip -> Builds parallel hip optimized version of the library
make hipdebug -> Builds serial hip debug version of the library
make phipdebug -> Builds parallel hip debug version of the library
Note that any of the above shortcuts accept configuration options, either at the
command line or through a user configuration file.
The build can be quick-tested by running
make check
which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:
make test
Note that by default MFEM uses "mpirun -np" in its test runs (this is also what
is used in the sample runs of its examples and miniapps). The MPI launcher can
be changed by the user as described in the "Specifying an MPI job launcher"
section at the end of this file.
Running all the tests may take a while. Implementation details about the check
and test targets can be found in the top-level makefile and the config/test.mk
file.
An optional installation of the library and the headers can be performed with
make install [PREFIX=<dir>]
The library will be installed in $(PREFIX)/lib, the headers in
$(PREFIX)/include, and the configuration makefile (config.mk) in
$(PREFIX)/share/mfem. The PREFIX option can also be set during configuration.
Information about the current build configuration can be viewed using
make status
make info
To clean the library and object files, but keep the current configuration, use
make clean
To clean everything, including the current configuration, use
make distclean
For a short help message, use
make help
The build process creates the MFEM library (libmfem.a) and the include file
(mfem.hpp) needed in MFEM-based applications, see e.g. the example codes in the
examples/ directory or the miniapps in the miniapps/ directory. A selected
subset of configuration options and derived makefile variables are also written
to the file config/config.mk. This file can be included by other makefiles to
obtain information about the MFEM configuration, see e.g. the makefile in the
examples/ directory.
Configuration options (GNU make)
================================
See the configuration file config/defaults.mk for the default settings.
Compilers:
CXX - C++ compiler, serial build
MPICXX - MPI C++ compiler, parallel build
CUDA_CXX - The CUDA compiler, 'nvcc'
Compiler options:
OPTIM_FLAGS - Options for optimized build
DEBUG_FLAGS - Options for debug build
CXXFLAGS - If not set, defined based on the above optimized/debug flags
CPPFLAGS - Additional compiler options
Build options:
STATIC - Build a static version of the library (YES/NO), default = YES
SHARED - Build a shared version of the library (YES/NO), default = NO
Installation options:
PREFIX - Specify the installation directory. The library (libmfem.a) will be
installed in $(PREFIX)/lib, the headers in $(PREFIX)/include, and
the configuration makefile (config.mk) in $(PREFIX)/share/mfem.
INSTALL - Specify the install program, e.g /usr/bin/install
MFEM library features/options (GNU make)
----------------------------------------
MFEM_USE_MPI = YES/NO
Choose parallel/serial build. The parallel build requires proper setup of the
HYPRE_* and METIS_* library options, see below.
MFEM_USE_METIS = YES/NO
Enable/disable the use of the METIS library. By default, this option is set
to the value of MFEM_USE_MPI. If this option is explicitly disabled in a
parallel build, then the only parallel partitioning (domain decomposition)
option in the library will be Cartesian partitioning with box meshes, and
thus most of the parallel examples and miniapps will fail.
MFEM_PRECISION = double/Double/DOUBLE/single/Single/SINGLE
Use single (float type) or double floating-point precision. In the
configuration header 'config/_config.hpp' this option is represented by
defining exactly one of the macros: MFEM_USE_DOUBLE, or MFEM_USE_SINGLE.
In the exported config files 'config.mk' and 'MFEMConfig.cmake', the option
is represented by the variables MFEM_USE_DOUBLE and MFEM_USE_SINGLE defined
as YES/NO (make) or ON/OFF (cmake). For more details see
https://github.com/orgs/mfem/discussions/4207
MFEM_DEBUG = YES/NO
Choose debug/optimized build. The debug build enables a number of messages
and consistency checks that may simplify bug-hunting.
MFEM_USE_EXCEPTIONS = YES/NO
Enable the use of exceptions. In particular, modifies the default behavior
when errors are encountered: throw an exception, instead of aborting.
MFEM_USE_LIBUNWIND = YES/NO
Use libunwind to print a stacktrace whenever mfem_error is raised. The
information printed is enough to determine the line numbers where the
error originated, provided MFEM_DEBUG=YES or build flags include `-g'.
MFEM_USE_METIS_5 = YES/NO
Specify the version of the METIS library - 5 (YES) or 4 (NO).
MFEM_USE_LAPACK = YES/NO
Use LAPACK routines for various dense linear algebra operations. When
enabled, this option uses the LAPACK_* library options, see below. (When not
enabled MFEM provides simple internal implementations where appropriate.)
MFEM_THREAD_SAFE = YES/NO
Use thread-safe implementation for some classes/methods. This comes at the
cost of extra memory allocation and de-allocation.
MFEM_USE_LEGACY_OPENMP = YES/NO
Enable (basic) experimental OpenMP support. Requires MFEM_THREAD_SAFE.
This option is deprecated.
MFEM_USE_OPENMP = YES/NO
Enable the OpenMP backend.
MFEM_USE_MEMALLOC = YES/NO
Internal MFEM option: enable batch allocation for some small objects.
Recommended value is YES.
MFEM_TIMER_TYPE = 0/1/2/3/4/5/6/NO
Specify which library functions to use in the class StopWatch used for
measuring time. The available options are:
0 - use std::clock from <ctime>, standard C++
1 - use times from <sys/times.h>
2 - use high-resolution POSIX clocks (see option POSIX_CLOCKS_LIB)
3 - use QueryPerformanceCounter from <windows.h>
4 - use mach_absolute_time from <mach/mach_time.h> + std::clock (Mac)
5 - use gettimeofday from <sys/time.h>
6 - use MPI_Wtime from <mpi.h>
NO - use option 3 if the compiler macro _WIN32 is defined, 0 otherwise
MFEM_USE_SUNDIALS = YES/NO
Enable MFEM time integrators and non-linear solvers based on the SUNDIALS
library. When enabled, this option uses the SUNDIALS_* library options,
see below.
MFEM_USE_SUITESPARSE = YES/NO
Enable MFEM functionality based on the SuiteSparse library. Currently, this
option adds the classes UMFPackSolver and KLUSolver (both sparse serial
direct solvers). When enabled, this option uses the SUITESPARSE_* library
options, see below.
MFEM_USE_SUPERLU = YES/NO
Enable MFEM functionality based on the SuperLU_DIST library. Currently, this
option adds the classes SuperLUSolver (a parallel sparse direct solver) and
SuperLURowLocMatrix a distributed CSR matrix class needed by SuperLU. When
enabled, this option uses the SUPERLU_* library options, see below.
MFEM_USE_SUPERLU5 = YES/NO
If SuperLU functionality is enabled, use the older 5.1.0 version rather than
the more recent 6+ versions.
MFEM_USE_MUMPS = YES/NO
Enable MFEM functionality based on the MUMPS library. Currently, this
option adds the class MUMPSSolver (a parallel sparse direct solver).
When enabled, this option uses the MUMPS_* library options, see below.
MFEM_USE_STRUMPACK = YES/NO
Enable MFEM functionality based on the STRUMPACK sparse direct solver and
preconditioner through the STRUMPACKSolver and STRUMPACKRowLocMatrix
classes. When enabled, this option uses the STRUMPACK_* library options, see
below.
MFEM_USE_GINKGO = YES/NO
Enable MFEM functionality based on the Ginkgo library, which provides
iterative linear solvers and preconditioners with OpenMP, CUDA backends, see
https://github.com/ginkgo-project/ginkgo. When enabled, the user can use
Ginkgo's solvers and preconditioners as shown in examples/ginkgo/.
MFEM_USE_AMGX = YES/NO
Enable MFEM functionality based on the AmgX multigrid library from NVIDIA.
Allows the user to use SparseMatrices and HypreParMatrices to solve linear
systems with the routines from the AmgX library.
MFEM_USE_GNUTLS = YES/NO
Enable secure socket support in class socketstream, using the auxiliary
GnuTLS_* classes, based on the GnuTLS library. This option may be useful in
multi-user environment to prevent users from sending/receiving visualization
data to/from other users. When this option is enabled, the default behavior
in class socketstream is to use secure sockets, e.g. when connecting to a
GLVis visualization server. In order for this to work, one needs to generate
GLVis server/client key pairs (in ~/.config/glvis), similar to ssh keys --
the script 'glvis-keygen.sh' in the main GLVis directory can be used to do
that:
bash glvis-keygen.sh ["Your Name"] ["Your Email"]
In MFEM v3.3.2 and earlier, the secure authentication is based on OpenPGP
keys, while later versions use X.509 certificates. The latest version of the
script 'glvis-keygen.sh' can be used to generate both types of keys.
When MFEM_USE_GNUTLS is enabled, the additional build options, GNUTLS_*, are
also used, see below.
MFEM_USE_NETCDF = YES/NO
NetCDF is the library that is used by the SNL Cubit mesh generator to create
Genesis mesh files. This option enables a reader for these files, which
requires that NetCDF be installed, see the NETCDF_* build options below.
MFEM_USE_PETSC = YES/NO
Enable MFEM linear and non-linear solvers, preconditioners, time integrators
and other features based on the PETSc package. When enabled, this option uses
the PETSC_* library options, see below.
MFEM_USE_SLEPC = YES/NO
Enable MFEM eigensolvers based on the SLEPc package. When enabled, this
option uses the SLEPC_* library options, see below.
MFEM_USE_MPFR = YES/NO
MPFR is a library for multiple-precision floating-point computations. This
option enables the use of MPFR in MFEM, e.g. for precise computation of 1D
quadrature rules. When enabled, this option uses the MPFR_* library options,
see below.
MFEM_USE_SIDRE = YES/NO
Sidre is a component of LLNL's axom project, https://github.com/LLNL/axom,
that provides an HDF5-based file format for visualization or restart
capability following the Conduit (https://github.com/LLNL/conduit) mesh
blueprint specification. When enabled, this option requires installation of
HDF5 (see also MFEM_USE_NETCDF), Conduit and LLNL's axom project.
MFEM_USE_SIMD = YES/NO
Enables the high performance templated classes to use architecture dependent
SIMD intrinsics instead of the generic implementation of class AutoSIMD in
linalg/simd/auto.hpp. This option should be combined with suitable
compiler options, such as -march=native, to enable optimal vectorization.
MFEM_USE_CONDUIT = YES/NO
Enables support for converting MFEM Mesh and Grid Function objects to and
from Conduit Mesh Blueprint Descriptions (https://github.com/LLNL/conduit/)
and support for JSON and Binary I/O via Conduit Relay. This option requires
an installation of Conduit. If Conduit was built with HDF5 support, it also
requires an installation of HDF5 (see also MFEM_USE_NETCDF).
MFEM_USE_ADIOS2 = YES/NO
Enables support for ADIOS2, version 2 of the adaptable input output system
for scientific data management. In MFEM, ADIOS2 provides parallel I/O with
ParaView visualization.
MFEM_USE_ZLIB = YES/NO
Enables use of on-the-fly gzip compressed streams. With this feature enabled
(YES), MFEM can compress its output files on-the-fly. In addition, it can
read back files compressed with zlib (or any compression utility capable
of creating a gzip-compatible output such as gzip).
MFEM will write compressed files if the mode argument in the constructor
includes a 'z' character. With this feature disabled (NO), MFEM will not be
able to properly read an input file if it is gzip compressed. In that case,
the solution is to uncompress the file with an external tool (such as gunzip)
before attempting to use it with MFEM.
When enabled, this option uses the ZLIB_* library options, see below.
MFEM_USE_PUMI = YES/NO
Enable the usage of PUMI (https://scorec.rpi.edu/pumi/) in MFEM. The Parallel
Unstructured Mesh Infrastructure (PUMI) is an unstructured, distributed mesh
data management system that is capable of handling general non-manifold
models and effectively supports automated adaptive analysis. PUMI enables
support for parallel unstructured mesh modifications in MFEM.
The develop branch of PUMI repository (https://github.com/SCOREC/core)
should be used for most updated features.
MFEM_USE_UMPIRE = YES/NO
Enables support for Umpire, a resource management library that allows the
discovery, provision, and management of memory on machines with multiple
memory devices like NUMA and GPUs.
MFEM_USE_BENCHMARK = YES/NO
Enables support for Google Benchmark, a library to support the benchmarking
of functions, in the tests/benchmarks directory.
MFEM_USE_HIOP = YES/NO
Enable the usage of HiOp (https://github.com/LLNL/hiop) in MFEM. HiOp is an
HPC solver for nonlinear optimization problems.
MFEM_USE_CODIPACK = YES/NO
Enable automatic differentiation using the CoDiPack library.
www.scicomp.uni-kl.de/codi/
MFEM_USE_ALGOIM = YES/NO
Enable the usage of Algoim - a collection of high-order accurate numerical
methods and C++ algorithms for working with implicitly-defined geometry and
level set methods. The Algoim library requires the Blitz++ library. The MFEM
provides interface to Algoim v1. Thus, to check out the specific state use:
git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a
https://algoim.github.io
MFEM_USE_ADFORWARD = YES/NO
Enable forward mode for AD packages. This option is valid
only if the AD package supports two modes (backward/forward).
MFEM_USE_CUDA = YES/NO
Enables support for CUDA devices in MFEM. CUDA is a parallel computing
platform and programming model for general computing on graphical processing
units (GPUs). The variable CUDA_ARCH is used to specify the CUDA compute
capability used during compilation (by default, CUDA_ARCH=sm_60). When
enabled, this option uses the CUDA_* build options, see below.
MFEM_USE_HIP = YES/NO
Enables support for AMD devices in MFEM. HIP is a heterogeneous-compute
interface for portability developed by AMD that can target both AMD and
NVIDIA GPUs. The variable HIP_ARCH is used to specify the AMD GPU processor
used during compilation (by default, HIP_ARCH=gfx900). When enabled, this
option uses the HIP_* build options, see below.
MFEM_USE_RAJA = YES/NO
Enable support for the RAJA performance portability layer in MFEM. RAJA
provides a portable abstraction for loops, supporting different programming
model backends. When using RAJA built with CUDA support, CUDA support must be
also enabled in MFEM, i.e. MFEM_USE_CUDA=YES must be set.
MFEM_USE_OCCA = YES/NO
Enables support for the OCCA library in MFEM. OCCA is an open-source library
which aims to make it easy to program different types of devices (e.g. CPU,
GPU, FPGA) by providing an unified API for interacting with JIT-compiled
backends. In order to use the OCCA CUDA backend, CUDA support must be enabled
in MFEM as well, i.e. MFEM_USE_CUDA=YES must be set.
MFEM_USE_GSLIB = YES/NO
Enables MFEM functionality based on the GSLIB library, and specifically its
FindPoints component, which provides a robust algorithms to evaluate finite
element functions in a collection of points in physical space. When enabled,
the user can use the GSLIB-FindPoints methods as shown in miniapps/gslib.
MFEM_USE_CEED = YES/NO
Enables support for the libCEED library in MFEM. libCEED is a portable
library for performant high-order operator evaluation developed by the Center
for Efficient Exascale Discretizations in the Exascale Computing Project.
MFEM_USE_MKL_CPARDISO = YES/NO
Enables the interface to MKL CPardiso: the Intel MKL Parallel Direct Sparse
Solver for Clusters. Make sure to set the correct values for MKL_MPI_WRAPPER
and MKL_LIBRARY_SUBDIR as shown in defaults.mk. If you configure MFEM with
MFEM_USE_LAPACK=YES, verify that the MKL LAPACK libraries are used. The
OpenMP capabilities are disabled at link time.
MFEM_USE_MOONOLITH = YES/NO
Enables the ParMoonolith interface for parallel non-conforming, non-matching,
variational, volumetric mesh information transfer. It requires the variable
MOONOLITH_DIR=<path to installation> to be defined in the environment in
order to be used with the Makefile. Makefile users are also required to
install moonolith using the command `make install_all`, see
https://bitbucket.org/zulianp/par_moonolith for details.
Although Moonolith is an MPI-based library, both serial (MFEM_USE_MPI=NO) and
parallel (MFEM_USE_MPI=YES) versions of MFEM are supported.
MFEM_USE_CALIPER = YES/NO
Enables the interface to Caliper. Caliper is a library to integrate
performance profiling capabilities into applications. To use Caliper,
developers mark code regions of interest using either Caliper's annotation
API or their equivalent in MFEM. Applications can then enable performance
profiling at runtime with Caliper's configuration API. Alternatively, one
can configure Caliper through environment variables or config files.
MFEM_USE_FMS = YES/NO
Enables support for the FMS library which consists of the DataCollection
sub-class mfem::FMSDataCollection for I/O in FMS formats, see the header file
fem/fmsdatacollection.hpp. In addition, this option enables in-memory
conversion routines between FMS's FmsDataCollection structure and MFEM's
DataCollection class, see the header file fem/fmsconvert.hpp.
MFEM_USE_PARELAG = YES/NO
Enables the miniapps that use the ParELAG library. MFEM does not currently
use ParELAG. In fact, ParELAG is dependent on MFEM. Therefore, this option
currently only concerns the miniapps.
MFEM_USE_TRIBOL = YES/NO
Enables the miniapps that use the Tribol library. MFEM does not currently
use Tribol. In fact, Tribol is dependent on MFEM. Therefore, this option
currently only concerns the miniapps.
MFEM_USE_ENZYME = YES/NO
Enables automatic differentiation support through the LLVM plugin Enzyme.
This requires the compiler to be set to clang (>=14.0.0). We also advise to
use the link time optimization (LTO) plugin, to enable functions that you
define over multiple files (compilation units) and want to be differentiated
automatically, to work. This requires to also use LLVM/LLD for linking.
Recommended options are in config/defaults.mk.
MFEM_BUILD_TAG = (any value)
An optional tag to characterize the build. Exported to config/config.mk.
Can be used to identify the MFEM build from other makefiles.
VERBOSE = YES/NO
Print some informational messages when building.
External libraries (GNU make):
------------------------------
Two types of library configuration options are used:
<LIBNAME>_OPT - for compiler options which usually specify an include path,
e.g.: -I/home/user/hypre/include
<LIBNAME>_LIB - for link options which usually specify link path and library
name, e.g.: -L/home/user/hypre/lib -lHYPRE
If specifying relative paths, they should be relative to the top-level MFEM
directory and use the string @MFEM_DIR@, e.g. HYPRE_OPT = -I@MFEM_DIR@/../hypre.
The specific libraries and their options are:
- HYPRE, required for the parallel build, i.e. when MFEM_USE_MPI = YES.
See also the "Specific options for hypre" section at the end of this file.
URL: https://github.com/hypre-space/hypre and https://www.llnl.gov/casc/hypre
Options: HYPRE_OPT, HYPRE_LIB.
Versions: HYPRE >= 2.10.0b (HYPRE built without CUDA)
HYPRE >= 2.20.0 (HYPRE built with '--enable-mixedint')
HYPRE >= 2.22.1 (HYPRE built with CUDA)
HYPRE >= 2.23.0 (HYPRE built with HIP)
HYPRE >= 2.31.0 (runtime selectable HYPRE execution on CPU/GPU)
- METIS, used when MFEM_USE_METIS = YES. If using METIS 5, set
MFEM_USE_METIS_5 = YES (default is to use METIS 4). For building instructions,
see the following:
- METIS 4.0.3: https://mfem.org/building/#parallel-mpi-version-of-mfem
- METIS 5.1.0: https://mfem.org/building/#parallel-build-using-metis-5
URL: https://github.com/mfem/tpls (MFEM mirror, see above)
Options: METIS_OPT, METIS_LIB.
Versions: METIS 4.0.3 or 5.1.0.
- LAPACK (optional), used when MFEM_USE_LAPACK = YES. Alternative, optimized
implementations can also be used, e.g. the ATLAS project.
URL: http://www.netlib.org/lapack (LAPACK)
http://math-atlas.sourceforge.net (ATLAS)
Options: LAPACK_OPT (currently not used/needed), LAPACK_LIB.
- OpenMP (optional), usually part of compiler, used when either MFEM_USE_OPENMP
or MFEM_USE_LEGACY_OPENMP is set to YES.
Options: OPENMP_OPT, OPENMP_LIB.
- High-resolution POSIX clocks: when using MFEM_TIMER_TYPE = 2, it may be
necessary to link with a system library (e.g. librt.so).
Option: POSIX_CLOCKS_LIB (default = -lrt).
- SUNDIALS (optional), used when MFEM_USE_SUNDIALS = YES.
Beginning with MFEM v3.3, SUNDIALS v2.7.0 is supported.
Beginning with MFEM v3.3.2, SUNDIALS v3.0.0 is also supported.
Beginning with MFEM v4.1, only SUNDIALS v5.0.0+ is supported.
When MFEM_USE_CUDA is enabled, only SUNDIALS v5.4.0+ is supported.
If MFEM_USE_MPI is enabled, we expect that SUNDIALS is built with support for
both MPI and hypre.
If MFEM_USE_CUDA is enabled, we expect that SUNDIALS is built with support
for CUDA.
If MFEM_USE_HIP is enabled, we expect that SUNDIALS is built with support
for HIP.
URL: http://computing.llnl.gov/projects/sundials/sundials-software
Options: SUNDIALS_OPT, SUNDIALS_LIB.
Versions: SUNDIALS >= 5.0.0,
SUNDIALS >= 5.4.0 for CUDA support, and
SUNDIALS >= 5.7.0 for HIP support.
- SuiteSparse (optional), used when MFEM_USE_SUITESPARSE = YES.
URL: http://faculty.cse.tamu.edu/davis/suitesparse.html
Options: SUITESPARSE_OPT, SUITESPARSE_LIB.
Versions: SuiteSparse >= 4.5.4, older versions may work too.
- SuperLU_DIST (optional), used when MFEM_USE_SUPERLU = YES. Note that
SuperLU_DIST requires ParMETIS, which includes METIS 5 in its distribution.
Both ParMETIS and the included METIS 5 should be built and installed in the
same location. If using SuperLU_Dist v5, set MFEM_USE_SUPERLU5=YES.
URL: http://crd-legacy.lbl.gov/~xiaoye/SuperLU
Options: SUPERLU_OPT, SUPERLU_LIB.
Versions: SuperLU_DIST >= 5.1.0.
- MUMPS (optional), used when MFEM_USE_MUMPS = YES. Note that MUMPS
requires LAPACK, SCALAPACK and a reordering package such as PORD or METIS.
URL: http://mumps.enseeiht.fr
Options: MUMPS_OPT, MUMPS_LIB.
Versions: MUMPS >= 5.1.1
- STRUMPACK (optional), used when MFEM_USE_STRUMPACK = YES. Note that STRUMPACK
requires the PT-Scotch and Scalapack libraries as well as ParMETIS, which
includes METIS 5 in its distribution. Starting with STRUMPACK v2.2.0, ParMETIS
and PT-Scotch are optional dependencies.
The support for STRUMPACK was added in MFEM v3.3.2.
URL: http://portal.nersc.gov/project/sparse/strumpack
Options: STRUMPACK_OPT, STRUMPACK_LIB.
Versions: STRUMPACK >= 3.0.0.
- Ginkgo (optional), used when MFEM_USE_GINKGO = YES. Note that Ginkgo needs a
C++ compiler that supports the C++-14 standard. For additional requirements
and dependencies of specific modules, see the Ginkgo webpage below.
URL: https://ginkgo-project.github.io
Options: GINKGO_OPT, GINKGO_LIB, GINKGO_DIR, GINKGO_BUILD_TYPE (Release or
Debug).
Versions: Ginkgo >= 1.4.0.
- AmgX (optional), used when MFEM_USE_AMGX = YES.
URL: https://github.com/NVIDIA/AMGX
Options: AMGX_OPT, AMGX_LIB.
Versions: AmgX >= 2.1, older versions may work too.
- GnuTLS (optional), used when MFEM_USE_GNUTLS = YES. On most Linux systems,
GnuTLS is available as a development package, e.g. gnutls-devel. On Mac OS X,
one can get the library through the Homebrew package manager (http://brew.sh).
URL: http://gnutls.org
Options: GNUTLS_OPT, GNUTLS_LIB.
Versions: GnuTLS >= 2.12.0, older versions may work too.
- NetCDF (optional), used when MFEM_USE_NETCDF = YES, required for reading Cubit
mesh files. Also requires installation of HDF5 and ZLIB, as explained at the
NetCDF web site. Note that we use the plain vanilla "C" version of NetCDF, you
don't need the C++ or parallel versions.
URL: www.unidata.ucar.edu/software/netcdf
Options: NETCDF_OPT, NETCDF_LIB.
Versions: NetCDF >= 4.4.0.
- PETSc (optional), used when MFEM_USE_PETSC = YES. Version 3.21 or higher of
the PETSC dev branch is required, though depending on the functionality older
versions may work too. The MFEM and PETSc builds can share common libraries,
e.g., hypre and SUNDIALS. Here's an example configuration, assuming
PETSc has been cloned on the same level as mfem and hypre:
./configure --download-fblaslapack=yes --download-scalapack=yes \
--download-mumps=yes --download-suitesparse=yes \
--with-hypre-dir=../hypre/src/hypre \
--with-shared-libraries=0
When building PETSc with HIP, one may need to add a flag like -std=c2x to
CFLAGS to allow proper parsing of the hipsparse header under C.
URL: https://www.mcs.anl.gov/petsc
Options: PETSC_OPT, PETSC_LIB.
Versions: PETSc >= 3.21.0, older versions may work too.
- SLEPc (optional), used when MFEM_USE_SLEPC = YES. SLEPc depends on PETSc and
uses some of the PETSc options when compiled.
URL: https://slepc.upv.es/
Options: SLEPC_OPT, SLEPC_LIB.
Versions: SLEPc >= 3.8.0.
- Sidre (optional), part of LLNL's axom project, used when MFEM_USE_SIDRE = YES.
Starting with MFEM v4.1, Axom version 0.3.1 or later is required.
URL: https://github.com/LLNL/axom
https://github.com/LLNL/conduit (Conduit)
https://support.hdfgroup.org/HDF5 (HDF5)
Options: SIDRE_OPT, SIDRE_LIB.
Versions: Axom >= 0.3.1.
- Conduit (optional), used when MFEM_USE_CONDUIT = YES. Conduit Mesh Blueprint
support requires Conduit >= v0.3.1 and VisIt >= v2.13.1 to read the output.
URL: https://github.com/LLNL/conduit (Conduit)
https://support.hdfgroup.org/HDF5 (HDF5)
Options: CONDUIT_OPT, CONDUIT_LIB.
Versions: Conduit >= 0.3.1.
- ADIOS2 (optional) used when MFEM_USE_ADIOS2 = YES.
URL: https://adios2.readthedocs.io/
Versions: ADIOS >= 2.5.0.
- PUMI (optional), used when MFEM_USE_PUMI = YES.
URL: https://scorec.rpi.edu/pumi
https://github.com/SCOREC/core
Options: PUMI_OPT, PUMI_LIB.
Versions: PUMI >= 2.2.6.
- HiOp (optional), used when MFEM_USE_HIOP = YES.
URL: https://github.com/LLNL/hiop
Options: HIOP_OPT, HIOP_LIB.
Versions: HIOP >= 0.4.6.
- CoDiPack (optional), used with MFEM_USE_CODIPACK = YES
URL: https://www.scicomp.uni-kl.de/codi/
Options: CODIPACK_OPT
Versions: 1.9.3
- GSLIB (optional), used when MFEM_USE_GSLIB = YES. The gslib library must be
built prior to the MFEM build, as follows: download gslib-1.0.7, untar it at
the same level as MFEM and create a symbolic link: "ln -s gslib-1.0.7 gslib".
Build gslib in parallel or in serial based on the desired MFEM build: "make
clean; make CC=mpicc" or "make clean; make CC=gcc MPI=0". Build MFEM with
MFEM_USE_GSLIB=YES.
URL: https://github.com/gslib/gslib/archive/v1.0.7.tar.gz
Options: GSLIB_OPT, GSLIB_LIB.
Versions: GSLIB >= 1.0.7.
- ALGOIM (optional), used when MFEM_USE_ALGOIM=YES. The library provides only
headers so it just needs to be downloaded at the same level as MFEM. Download
the specific version we use as:
"git clone https://github.com/algoim/algoim.git;
git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a"
ALGOIM depends on BLITZ and the library must be built prior to the MFEM build.
Download v1.0.2, untar it at the same level as MFEM and create a symbolic link:
"ln -s blitz-1.0.2 blitz".
Build Blitz using CMake as:
"cmake . -DCMAKE_INSTALL_PREFIX=.; make lib; make install"
URL: https://github.com/blitzpp/blitz/archive/refs/tags/1.0.2.tar.gz
Options: BLITZ_OPT, BLITZ_LIB
Versions: BLITZ = 1.0.2
- MKL CPardiso (optional), used when MFEM_USE_MKL_CPARDISO = YES.
URL: https://software.intel.com/content/www/us/en/develop/tools/math-kernel-library.html
Options: MKL_CPARDISO_OPT, MKL_CPARDISO_LIB.
Versions: Intel MKL >= 2020.
- CUDA (optional), used when MFEM_USE_CUDA = YES.
URL: https://developer.nvidia.com/cuda-toolkit
Options: CUDA_CXX, CUDA_ARCH, CUDA_OPT, CUDA_LIB.
Versions: CUDA >= 10.1.168.
- HIP (optional), used when MFEM_USE_HIP = YES.
URL: https://rocmdocs.amd.com
Options: HIP_CXX, HIP_ARCH, HIP_OPT, HIP_LIB.
- OCCA (optional), used when MFEM_USE_OCCA = YES.
URL: https://libocca.org
Options: OCCA_DIR, OCCA_OPT, OCCA_LIB.
Versions: OCCA >= 1.1.0.
- libCEED (optional), used when MFEM_USE_CEED = YES.
URL: https://github.com/CEED/libCEED
https://ceed.exascaleproject.org/libceed
Options: CEED_DIR, CEED_OPT, CEED_LIB.
Versions: libCEED >= 0.12.
- RAJA (optional), used when MFEM_USE_RAJA = YES.
Beginning with MFEM v4.5.1, only RAJA v2022.10.3+ is supported.
URL: https://github.com/LLNL/RAJA
Options: RAJA_DIR, RAJA_OPT, RAJA_LIB.
Versions: RAJA >= 2022.10.3.
- Moonolith (optional), use when MFEM_USE_MOONOLITH = YES.
URL: https://bitbucket.org/zulianp/par_moonolith
Options: MOONOLITH_DIR
Versions: MOONOLITH >= 1.1.0.
- Caliper (optional), used when MFEM_USE_CALIPER = YES.
URL: https://github.com/LLNL/Caliper
Options: CALIPER_DIR
Versions: CALIPER >= 2.5.0, older versions may work too.
- Umpire, used when MFEM_USE_UMPIRE = YES.
Umpire requires camp when the Umpire version is >= 3.0.0.
URL: https://github.com/LLNL/Umpire
Options: UMPIRE_DIR, UMPIRE_OPT, UMPIRE_LIB.
Versions: Umpire >= 3.0.0.
- Benchmark, used when MFEM_USE_BENCHMARK = YES.
URL: https://github.com/google/benchmark
Options: BENCHMARK_DIR, BENCHMARK_LIB.
Versions: Benchmark >= 1.5.6.
- MPFR (optional), used when MFEM_USE_MPFR = YES.
URL: http://mpfr.org, it depends on the GMP library: https://gmplib.org
Options: MPFR_OPT, MPFR_LIB.
- Libunwind (optional), used when MFEM_USE_LIBUNWIND = YES. The library is
included with OS X (as of version 10.11). On Linux it could be installed with
the libunwind-devel package.
URL: http://www.nongnu.org/libunwind
Options: LIBUNWIND_OPT, LIBUNWIND_LIB.
- ZLIB (optional), used when MFEM_USE_ZLIB = YES, or when MFEM_USE_NETCDF =
YES (in the default settings for NETCDF_OPT and NETCDF_LIB).
URL: https://zlib.net
Options: ZLIB_OPT, ZLIB_LIB.
- FMS (optional), used when MFEM_USE_FMS = YES.
URL: https://github.com/CEED/FMS
Options: FMS_OPT, FMS_LIB.
Versions: FMS >= 0.2.
- ParELAG, used when MFEM_USE_PARELAG = YES.
URL: https://github.com/LLNL/parelag
Options: PARELAG_DIR, PARELAG_OPT, PARELAG_LIB.
- Tribol, used when MFEM_USE_TRIBOL = YES.
URL: https://github.com/LLNL/Tribol
Options: TRIBOL_DIR, TRIBOL_OPT, TRIBOL_LIB.
- Enzyme, used when MFEM_USE_ENZYME = YES. Requires LLVM/Clang >= 14.0.0.
URL: https://github.com/EnzymeAD/Enzyme
Options: ENZYME_DIR, ENZYME_OPT, ENZYME_LIB.
Versions: Enzyme >= v0.0.33.
Building with CMake
===================
The MFEM build system consists of two steps: configuration and compilation.
The configuration step can be used to adjust paths to external libraries,
compilers, flags, etc, similar to any CMake build system. It is performed by
running
mkdir <mfem-build-dir> ; cd <mfem-build-dir>
cmake <mfem-source-dir> [OPTIONS] ...
The OPTIONS are of the form -D<VARIABLE>=<VALUE>, e.g. -DMFEM_USE_MPI=YES.
Detailed description of the configuration options is given below. Alternatively,
the options can be specified with an input file:
cd <mfem-source-dir>/config
cp defaults.cmake user.cmake
(edit user.cmake)
cd <mfem-build-dir>
cmake <mfem-source-dir>
Note that user.cmake, if present, is loaded before defaults.cmake (and thus the
former takes precedence over the latter) and its path/name can be changed with
cmake <mfem-source-dir> -DUSER_CONFIG=<user_config_file>
Debug and optimization options are controlled through the CMake variable
CMAKE_BUILD_TYPE which can be set to standard values like "Debug", and "Release"
(default).
To use a specific generator use the "-G <generator>" option of cmake:
cmake <mfem-source-dir> -G "Xcode"
cmake <mfem-source-dir> -G "Visual Studio 12 2013"
cmake <mfem-source-dir> -G "MinGW Makefiles"
With CMake it is possible to build MFEM as a shared library using the standard
CMake option -DBUILD_SHARED_LIBS=1.
Once configured, the library can be built simply with (assuming a UNIX type
system, where the default is to generate "UNIX Makefiles")
make -j 4
or
cmake --build .
or
cmake --build . --config Release [Visual Studio, Xcode]
The build can be quick-tested by running
make check
or
cmake --build . --target check
or
cmake --build . --config Release --target check [Visual Studio, Xcode]
which will simply compile and run Example 1/1p. For more extensive tests that
check the results from all the serial/parallel MFEM examples and miniapps use:
make exec -j 4
make test
or
cmake --build . --target exec
cmake --build . --target test
or
cmake --build . --config Release --target exec [Visual Studio, Xcode]
cmake --build . --config Release --target RUN_TESTS [Visual Studio, Xcode]
Note that running all the tests may take a while.
Installation prefix can be configured by setting the standard CMake variable
CMAKE_INSTALL_PREFIX. To install the library, use
make install
or
cmake --build . --target install
or
cmake --build . --config Release --target install [Xcode]
cmake --build . --config Release --target INSTALL [Visual Studio]
The library will be installed in <PREFIX>/lib, the headers in <PREFIX>/include,
and the configuration CMake files in <PREFIX>/lib/cmake/mfem.
Configuration variables (CMake)
===============================
See the configuration file config/defaults.cmake for the default settings.
Note: the option MFEM_USE_CUDA requires CMake version 3.8 or newer!
Non-standard CMake variables for compilers:
CXX - If set, overwrite the auto-detected C++ compiler, serial build
MPICXX - If set, overwrite the auto-detected MPI C++ compiler, parallel build
The compiler options for the various build types can be controlled using
standard CMake variables like CMAKE_CXX_FLAGS_RELEASE and CMAKE_CXX_FLAGS_DEBUG.
MFEM library features/options (CMake)
-------------------------------------
The following options are equivalent to the GNU make options with the same name:
[see "MFEM library features/options (GNU make)" above]
MFEM_USE_MPI
MFEM_USE_METIS - Set to ${MFEM_USE_MPI}, can be overwritten.
MFEM_PRECISION
MFEM_USE_LIBUNWIND
MFEM_USE_LAPACK
MFEM_THREAD_SAFE
MFEM_USE_LEGACY_OPENMP
MFEM_USE_OPENMP
MFEM_USE_MEMALLOC
MFEM_TIMER_TYPE - Set automatically, can be overwritten.
MFEM_USE_SUITESPARSE
MFEM_USE_SUPERLU
MFEM_USE_MUMPS
MFEM_USE_STRUMPACK
MFEM_USE_GINKGO
MFEM_USE_AMGX