Merge pull request #3199 from mfem/mfem-4.5-dev

Final changes for mfem-4.5

CHANGELOG

@@ -8,66 +8,64 @@
                                https://mfem.org
 
 
-Version 4.4.1 (development)
-===========================
+Version 4.5, released on October 22, 2022
+=========================================
 
 Meshing improvements
 --------------------
-- Added support for mixed meshes and pyramids in GSLIB-FindPoints.
-
 - Added new SubMesh and ParSubMesh classes that can be used to extract a subset
   of a given Mesh. These classes have the same functionality as Mesh and ParMesh
   and work with all existing MFEM interfaces like finite element spaces etc.
 
-- Added a new method Mesh::NodesUpdated, which should be called after the mesh
+- Added a method, ParMesh::GetSerialMesh(), that reconstructs a partitioned
+  parallel mesh on a given single rank. Also, added ParMesh::PrintAsSerial(),
+  which saves the reconstructed serial mesh to a C++ stream on rank 0.
+
+- Added more 3D TMOP metrics, as well as specialized metrics for mesh
+  untangling and worst-case quality improvement.
+
+- Added a new method, Mesh::NodesUpdated, which should be called after the mesh
   node coordinates have changed, e.g. after the mesh has moved. This is
   necessary, for example, with device assembly of linear and bilinear forms.
 
+- Added support for mixed meshes and pyramids in GSLIB-FindPoints.
+
 Discretization improvements
 ---------------------------
-- Added support for assembling low-order-refined matrices using a GPU-enabled
-  "batched" algorithm. The lor_solvers and plor_solvers now fully support GPU
-  acceleration with arbitrary user-supplied coefficients.
-
-- Added support for partial assembly and fully matrix-free operators on mixed
-  meshes (different element types and p-adaptivity) through libCEED, including
-  device acceleration, e.g. with NVIDIA and AMD GPUs. The p-adaptivity is
-  currently limited by MFEM capabilities, i.e. 2D serial meshes. All mixed
-  element topologies are supported in serial and parallel: segment, triangle,
-  square, tetrahedron, cube, prism, and pyramid.
-
 - Added full assembly and device support for several LinearForm integrators:
   * DomainLF: (f, v)
   * VectorDomainLF: ((f1,...,fn), (v1,...,vn))
   * DomainLFGrad: (f, grad(v))
   * VectorDomainLFGrad: ((f1x,f1y,f1z,...,fnx,fny,fnz), grad(v1,...,vn))
+  The device assembly of linear forms has to be explicitly enabled by calling
+  LinearForm::UseFastAssembly(true), otherwise the legacy linear form assembly
+  is used by default.
+
+- Added support for assembling low-order-refined matrices using a GPU-enabled
+  "batched" algorithm. The lor_solvers and plor_solvers now fully support GPU
+  acceleration with arbitrary user-supplied coefficients.
+
+- Added a new class FaceQuadratureSpace that allows for the construction of
+  QuadratureFunctions on the interior or boundary faces of a mesh.
 
-- The device assembly of linear forms now has to be explicitly enabled by
-  calling LinearForm::UseFastAssembly(true). By default, the legacy linear form
-  assembly is used, which was not the case in mfem-4.4. As part of this change,
-  the boolean parameter of the Assemble method of LinearForm has been removed.
+- Added a class CoefficientVector for efficient access of variable coefficient
+  values at quadrature points (in particular for GPU/device kernels).
 
 - Added WhiteGaussianNoiseDomainLFIntegrator: a LinearFormIntegrator class for
   spatial Gaussian white noise.
 
 - Added a new Zienkiewicz-Zhu patch recovery-based a posteriori error estimator.
   See fem/estimators.hpp.
 
-- Fixes and improvements in LinearFormExtension.
-
-- Added a new class FaceQuadratureSpace that allows for the construction of
-  QuadratureFunctions on the interior or boundary faces of a mesh.
-
-- Added a class CoefficientVector for efficient access of variable coefficient
-  values at quadrature points (in particular for GPU/device kernels).
+- Various fixes and improvements in LinearFormExtension.
 
 Linear and nonlinear solvers
 ----------------------------
-- Added a new class DGMassInverse that performs a local elementwise CG
+- Added a new class DGMassInverse that performs a local element-wise CG
   iteration to solve systems involving the discontinuous Galerkin mass matrix,
   including support for device/GPU acceleration.
 
-- Added more flexibility to the constraint solver classes:
+- Added more flexibility to the constrained solver classes:
   * PenaltyConstrainedSolver now allows for a vector of penalty parameters
     (necessary for penalty contact)
   * PenaltyConstrainedSolver and EliminationSolver can use GMRES or PCG
@@ -76,6 +74,9 @@ Linear and nonlinear solvers
  - Added functions to toggle additional options for the SuperLU_Dist and Hypre
    preconditioners (ParaSails, Euclid, ILU).
 
+- Added boundary elimination with device support for `SparseMatrix` and
+  `HypreParMatrix`.
+
 New and updated examples and miniapps
 -------------------------------------
 - Added a new elasticity miniapp, Hooke, that showcases a low-level approach of
@@ -84,57 +85,69 @@ New and updated examples and miniapps
   automatic differentiation tools like a native dual number implementation or a
   third party library such as Enzyme. See miniapps/elasticity for more details.
 
+- Added example for body-fitted volumetric and shape integration using the
+  Algoim library in miniapps/shifted.
+
 - Add a new example code, Example 33/33p, to demonstrate the solution of
   spectral fractional PDEs with MFEM.
 
 Integrations, testing and documentation
 ---------------------------------------
 - Added a Dockerfile for a simple MFEM container, see config/docker/README.md.
+  More sophisticated developer containers are available in the new repo
+  https://github.com/mfem/containers.
+
+- Added support for the LLVM-based automatic differentiation tool Enzyme, see
+  https://github.com/EnzymeAD/Enzyme. Build system flags and a convenience
+  header are provided. The functionality and interaction are demonstrated in a
+  new miniapp in miniapps/elasticity.
+
+- Added support for partial assembly and fully matrix-free operators on mixed
+  meshes (different element types and p-adaptivity) through libCEED, including
+  device acceleration, e.g. with NVIDIA and AMD GPUs. The p-adaptivity is
+  currently limited to 2D serial meshes. All mixed element topologies are
+  supported in both serial and parallel.
 
 - Added support for ParMoonolith, https://bitbucket.org/zulianp/par_moonolith,
   which provides parallel non-conforming, non-matching, variational, volumetric
   mesh information transfer. With ParMortarAssember, fields can be exchanged
   between arbitrarily distributed and unrelated finite element meshes in a
   variationally consistent way.
 
-- Added support for the LLVM-based automatic differentiation tool Enzyme, see
-  https://github.com/EnzymeAD/Enzyme. Build system flags and a convenience
-  header are provided. The functionality and interaction are demonstrated in a
-  new miniapp in miniapps/elasticity.
+- Fully encapsulated SUNDIALS `N_Vector` object within the `SundialsNVector`
+  class by removing deprecated (e.g. `HypreParVector::ToNVector`) and
+  non-deprecated (e.g. `Vector::ToNVector`) functions in other classes.
 
-- Added example for body-fitted volumetric and shape integration using the
-  Algoim library.
+- New benchmark for the different assembly levels inspired by the CEED
+  Bake-Off Problems, see tests/benchmarks/bench_assembly_levels.cpp.
 
 - Added Windows 2022 CI testing with GitHub actions.
 
 Miscellaneous
 -------------
-- Various other simplifications, extensions, and bugfixes in the code.
+- The method SparseMatrix::EnsureMultTranspose() is now automatically called
+  by the methods AddMultTranspose(), MultTranspose(), and AbsMultTranspose().
+  Added a method with the same name to class HypreParMatrix which is also called
+  automatically by the HypreParMatrix::MultTranspose() methods.
 
-- Added boundary elimination with device support for `SparseMatrix` and
-  `HypreParMatrix`.
+- Updated various MemoryUsage methods to return 'std::size_t' instead of 'long'
+  since the latter is 32-bit in Win64 builds.
 
 - When using `AssemblyLevel::FULL`, `FABilinearFormExtension::FormSystemMatrix`
   outputs an `OperatorHandle` containing a `SparseMatrix` in serial, and an
   `HypreParMatrix` in parallel (instead of a `ConstrainedOperator`).
 
-- Added more 3D TMOP metrics, as well as specialized metrics for mesh
-  untangling and worst-case quality improvement;
-
-- Fully encapsulated SUNDIALS `N_Vector` object within the `SundialsNVector`
-  class by removing deprecated (e.g. `HypreParVector::ToNVector`) and
-  non-deprecated (e.g. `Vector::ToNVector`) functions in other classes.
+- In various places in the library, replace the use of 'long' with 'long long'
+  to better support Win64 builds where 'long' is 32-bit and 'long long' is
+  64-bit. On Linux and MacOS, both types are typically 64-bit.
 
 - The behavior of GridFunction::GetTrueVector() has been changed to not return
   an empty true vector.
 
-- The method SparseMatrix::EnsureMultTranspose() is now automatically called
-  by the methods AddMultTranspose(), MultTranspose(), and AbsMultTranspose().
-  Added a method with the same name to class HypreParMatrix which is also called
-  automatically by the HypreParMatrix::MultTranspose() methods.
-
 - Added support for ordering search points byVDIM in FindPointsGSLIB.
 
+- Various other simplifications, extensions, and bugfixes in the code.
+
 
 Version 4.4, released on March 21, 2022
 =======================================
@@ -167,11 +180,6 @@ Meshing improvements
 - Added a simpler interface to access mesh face information, see FaceInformation
   and GetFaceInformation in the Mesh class.
 
-- Added the method ParMesh::GetSerialMesh() that reconstructs a partitioned
-  parallel mesh on a given single rank. Also, added the method
-  ParMesh::PrintAsSerial() that saves the reconstructed serial mesh to a C++
-  stream on rank 0.
-
 - Gmsh meshes where all elements have zero physical tag (the default Gmsh output
   format if no physical groups are defined) are now successfully loaded, and
   elements are reassigned attribute number 1.
@@ -261,9 +269,6 @@ Integrations, testing and documentation
 - Switched from Artistic Style (astyle) version 2.05.1 to version 3.1 for code
   formatting. See the "make style" target.
 
-- New benchmark for the different assembly levels inspired by the CEED
-  Bake-Off Problems, see tests/benchmarks/bench_assembly_levels.cpp.
-
 Miscellaneous
 -------------
 - Added a simple singleton class, Mpi, as a replacement for MPI_Session. New
@@ -274,13 +279,6 @@ Miscellaneous
 
 - Fixed several MinGW build issues on Windows.
 
-- In various places in the library, replace the use of 'long' with 'long long'
-  to better support Win64 builds where 'long' is 32-bit and 'long long' is
-  64-bit. On Linux and MacOS, both types are typically 64-bit.
-
-- Update various "MemoryUsage" methods to return 'std::size_t' instead of 'long'
-  since the latter is 32-bit in Win64 builds.
-
 - Added 'double' atomicAdd implementation for previous versions of CUDA.
 
 - HypreParVector and Vector now support C++ move semantics, and the copy

CMakeLists.txt

@@ -51,7 +51,7 @@ project(mfem NONE)
 # Current version of MFEM, see also `makefile`.
 #   mfem_VERSION = (string)
 #   MFEM_VERSION = (int)   [automatically derived from mfem_VERSION]
-set(${PROJECT_NAME}_VERSION 4.4.1)
+set(${PROJECT_NAME}_VERSION 4.5.0)
 
 # Prohibit in-source build
 if (${PROJECT_SOURCE_DIR} STREQUAL ${PROJECT_BINARY_DIR})
@@ -81,6 +81,10 @@ if (MFEM_USE_STRUMPACK)
   # Just needed to find the MPI_Fortran libraries to link with
   set(XSDK_ENABLE_Fortran ON)
 endif()
+# SUNDIALS >= 6.4.0 requires C++14:
+if (MFEM_USE_SUNDIALS AND ("${CMAKE_CXX_STANDARD}" LESS "14"))
+  set(CMAKE_CXX_STANDARD 14)
+endif()
 if (MFEM_USE_GINKGO AND ("${CMAKE_CXX_STANDARD}" LESS "14"))
   set(CMAKE_CXX_STANDARD 14)
 endif()
@@ -200,10 +204,10 @@ if (MFEM_USE_MPI)
   find_package(MPI REQUIRED)
   set(MPI_CXX_INCLUDE_DIRS ${MPI_CXX_INCLUDE_PATH})
   if (MFEM_MPIEXEC)
-    set(MPIEXEC ${MFEM_MPIEXEC})
+    string(REPLACE " " ";" MPIEXEC ${MFEM_MPIEXEC})
   endif()
   if (MFEM_MPIEXEC_NP)
-    set(MPIEXEC_NUMPROC_FLAG ${MFEM_MPIEXEC_NP})
+    string(REPLACE " " ";" MPIEXEC_NUMPROC_FLAG ${MFEM_MPIEXEC_NP})
   endif()
   # Parallel MFEM depends on hypre
   find_package(HYPRE REQUIRED)

INSTALL

@@ -7,6 +7,10 @@
 
                                https://mfem.org
 
+This file provides a detailed description of how to build and install the MFEM
+library. For a simple build, see the step-by-step instructions on the website
+at https://mfem.org/building.
+
 The MFEM library has a serial and an MPI-based parallel version, which largely
 share the same code base. The only prerequisite for building the serial version
 of MFEM is a (modern) C++ compiler, such as g++. The parallel version of MFEM
@@ -19,7 +23,7 @@ requires an MPI C++ compiler, as well as the following external libraries:
   https://github.com/mfem/tpls
 
   Note: We recommend our mirror of metis-4.0.3/5.1.0 above because the METIS
-  webpage, https://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down
+  webpage, http://glaros.dtc.umn.edu/gkhome/metis/metis/overview, is often down
   and we don't support yet the new repo https://github.com/KarypisLab/METIS.
 
 The hypre dependency can be downloaded as a tarball from GitHub or from the
@@ -758,12 +762,12 @@ The specific libraries and their options are:
   Options: GSLIB_OPT, GSLIB_LIB.
   Versions: GSLIB >= 1.0.7.
 
-- ALGOIM (optional), used when MFE_USE_ALGOIM=YES. The library provides only
+- ALGOIM (optional), used when MFEM_USE_ALGOIM=YES. The library provides only
   headers so it just needs to be downloaded at the same level as MFEM. Download
   the specific version we use as:
   "git clone https://github.com/algoim/algoim.git;
    git checkout 9c9ca0ef094d8ab0390ed36367a1151b459bbe0a"
-  ALGOIM depends on BLITZ and rhe library must be built prior to the MFEM build.
+  ALGOIM depends on BLITZ and the library must be built prior to the MFEM build.
   Download v1.0.2, untar it at the same level as MFEM and create a symbolic link:
   "ln -s blitz-1.0.2 blitz".
   Build Blitz using CMake as:

config/defaults.mk

@@ -252,12 +252,15 @@ POSIX_CLOCKS_LIB = -lrt
 # For sundials_nvecmpiplusx and nvecparallel remember to build with MPI_ENABLE=ON
 # and modify cmake variables for hypre for sundials
 SUNDIALS_DIR = @MFEM_DIR@/../sundials-5.0.0/instdir
+# SUNDIALS >= 6.4.0 requires C++14:
+ifeq ($(MFEM_USE_SUNDIALS),YES)
+   BASE_FLAGS = -std=c++14
+endif
 SUNDIALS_OPT = -I$(SUNDIALS_DIR)/include
 SUNDIALS_LIB = $(XLINKER)-rpath,$(SUNDIALS_DIR)/lib64\
  $(XLINKER)-rpath,$(SUNDIALS_DIR)/lib\
  -L$(SUNDIALS_DIR)/lib64 -L$(SUNDIALS_DIR)/lib\
  -lsundials_arkode -lsundials_cvodes -lsundials_nvecserial -lsundials_kinsol
-
 ifeq ($(MFEM_USE_MPI),YES)
    SUNDIALS_LIB += -lsundials_nvecparallel -lsundials_nvecmpiplusx
 endif

config/sample-runs.sh

@@ -554,15 +554,14 @@ function go()
    local cmd_line="${1##+( )}"
    cmd_line="${cmd_line%%+( )}"
    shopt -u extglob
-   eval local cmd=(${cmd_line})
    local res=""
    echo $sep
    echo "<${group}>" "${cmd_line}"
    echo $sep
    if [ "${timing}" == "yes" ]; then
-      timed_run "${cmd[@]}"
+      timed_run eval "${cmd_line}"
    else
-      "${cmd[@]}"
+      eval "${cmd_line}"
    fi
    if [ "$?" -eq 0 ]; then
       res="${green}  OK  ${none}"

config/vcpkg/ports/metis-mfem/vcpkg.json

@@ -3,5 +3,5 @@
   "version-string": "5.1.0",
   "port-version": 0,
   "description": "Serial Graph Partitioning and Fill-reducing Matrix Ordering",
-  "homepage": "https://glaros.dtc.umn.edu/gkhome/metis/metis/overview"
+  "homepage": "http://glaros.dtc.umn.edu/gkhome/metis/metis/overview"
 }

doc/CodeDocumentation.conf.in

@@ -38,7 +38,7 @@ PROJECT_NAME           = "MFEM"
 # could be handy for archiving the generated documentation or if some version
 # control system is used.
 
-PROJECT_NUMBER         = v4.4.1
+PROJECT_NUMBER         = v4.5.0
 
 # Using the PROJECT_BRIEF tag one can provide an optional one line description
 # for a project that appears at the top of each page and should give viewer a

fem/bilininteg.hpp

@@ -215,10 +215,10 @@ class BilinearFormIntegrator : public NonlinearFormIntegrator
                              function by any coefficients describing the
                              integrator.
        @param[in] ir  If passed (the default value is NULL), the implementation
-                        of the method will ignore the integration rule provided
-                        by the @a fluxelem parameter and, instead, compute the
-                        discrete flux at the points specified by the integration
-                        rule @a ir.
+                      of the method will ignore the integration rule provided
+                      by the @a fluxelem parameter and, instead, compute the
+                      discrete flux at the points specified by the integration
+                      rule @a ir.
     */
    virtual void ComputeElementFlux(const FiniteElement &el,
                                    ElementTransformation &Trans,