Skip to content

Commit

Permalink
Browse files Browse the repository at this point in the history
Inlcusion of high-order Nédélec elements
  • Loading branch information
ocastilloreyes committed Oct 9, 2018
1 parent a50def5 commit c98dcd2
Show file tree
Hide file tree
Showing 178 changed files with 5,715 additions and 999 deletions.
30 changes: 18 additions & 12 deletions DESCRIPTION.rst 100644 → 100755
@@ -1,9 +1,9 @@
PETGEM
petgem
======

PETGEM is a parallel python code for 3D Controlled-Source
Electromagnetic Method (3D CSEM) in geophysics using edge finite
elements (Nedelec finite elements).
Electromagnetic Method (3D CSEM) in geophysics using high-order edge finite
elements (Nédélec finite elements).


Requirements
Expand All @@ -29,15 +29,15 @@ Install

* Following commands may require root privileges

* Download `PETSc <https://www.mcs.anl.gov/petsc/>`__ (PETSc 3.7 and 3.8 have been tested)
* Download `PETSc <https://www.mcs.anl.gov/petsc/>`__ (PETSc 3.7, 3.8, and 3.9 have been tested)

* Uncompress the PETSc archive (in this example, using PETSc 3.8.3)::
* Uncompress the PETSc archive (in this example, using PETSc 3.9.3)::

$ tar zxvf petsc-3.8.3.tar.gz
$ tar zxvf petsc-3.9.3.tar.gz

* Configure and build PETSc. The configuration options depend on the calculations you want to perform (complex- or real-valued) as well as your compiler/MPI/Blas/Lapack setup. For PETGEM executions, **PETSC MUST BE BUILD FOR COMPLEX-VALUED NUMBERS**. In order to avoid incompatibilities between PETSC, petsc4py and PETGEM, we highly recommend the following configuration lines. Please, visit PETSc website for advanced configuration options. If you have a clean environment (not working MPI/Blas/Lapack), then run::

$ cd petsc-3.8.3
$ cd petsc-3.9.3
$ export PETSC_DIR=$PWD
$ export PETSC_ARCH=arch-linux2-c-debug

Expand All @@ -49,6 +49,12 @@ Install

$ --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-cmake

* Further, to activate GPUs support, please add following options to previous configure line::

$ --with-cuda=1 --with_cuda_dir=PATH

where ``PATH`` is the directory of your CUDA libraries.

* Then, build and install PETSc::

$ make $PETSC_DIR $PETSC_ARCH all
Expand Down Expand Up @@ -79,8 +85,8 @@ publication, please acknowledge that fact by citing the project:
elements*. Computers & Geosciences, vol 119: 123-136. ISSN 0098-3004,
Elsevier. https://doi.org/10.1016/j.cageo.2018.07.005

* Castillo-Reyes, O., de la Puente, J., Cela, J.M. (2017).
*Three-Dimensional CSEM Modelling on Unstructured Tetrahedral Meshes
Using Edge Finite Elements*. Communications in Computer and
Information Science, vol 697: 247-256. ISBN 978-3-319-57971-9,
Springer, Cham. https://doi.org/10.1007/978-3-319-57972-6_18
* Castillo-Reyes, O., de la Puente, J., Cela, J.M. (2017).
*Three-Dimensional CSEM Modelling on Unstructured Tetrahedral Meshes
Using Edge Finite Elements*, Communications in Computer and
Information Science, vol 697: 247-256. ISBN 978-3-319-57971-9
Springer, Cham. https://doi.org/10.1007/978-3-319-57972-6_18
6 changes: 3 additions & 3 deletions README.rst 100644 → 100755
Expand Up @@ -4,9 +4,9 @@ petgem
Overview
========

Welcome to petgem. This code is a parallel python modelling tool for 3D
Welcome to PETGEM. This code is a parallel python modelling tool for 3D
Controlled-Source Electromagnetic Method (3D CSEM) in geophysics using
edge finite elements (Nedelec finite elements).
high-order edge finite elements (Nédélec finite elements).

Dependencies
------------
Expand Down Expand Up @@ -34,7 +34,7 @@ License
PETGEM is developed as open-source under GPLv3.0 license at Computer
Applications in Science & Engineering of the Barcelona Supercomputing
Center - Centro Nacional de Supercomputación. Please, see the CONDITIONS
OF USE described in the LICENSE.rst file.
OF USE described in the LICENSE.rst file.


Documentation
Expand Down
Empty file modified doc/Makefile 100644 → 100755
Empty file.
58 changes: 30 additions & 28 deletions doc/source/CSEMEdges.rst 100644 → 100755
Expand Up @@ -19,8 +19,8 @@ of freedom, e.g. size of the problem. Furthermore, its divergence-free basis
is very well suited for solving Maxwell’s equation. On top of that, we choose
to support tetrahedral meshes, as these are the easiest to use for very large
domains or complex geometries. We present the numerical formulation and
results of 3D CSEM forward modelling (FM) using tetrahedral EFEM on
unstructured meshes.
results of 3D CSEM forward modelling (FM) using EFEM of high-order on
unstructured tetrahedral meshes.

.. _CSEM problem:

Expand All @@ -33,11 +33,11 @@ CSEM surveys generally work with low frequency electromagnetic fields
(:math:`\sim` 1 Hz to :math:`\sim` 3 Hz) because the electric conductivity of the geological
structures is much larger than their dielectric permittivity. As a
consequence, in an unbound domain :math:`\Gamma`, the electric field
can be obtained by solving Maxwell’s equations in their diffusive form:
can be obtained by solving Maxwell’s equations in their diffusive form

.. math::
\nabla \times \mathbf{E} &= i\omega \mu_{0}\mathbf H \\
\nabla \times \mathbf{H} &= \mathbf J_{s} + \tilde{\sigma}\mathbf E
\nabla \times \mathbf{E} &= i\omega \mu_{0}\mathbf H, \\
\nabla \times \mathbf{H} &= \mathbf J_{s} + \tilde{\sigma}\mathbf E,
:label: maxwellDiffusive
where the harmonic time dependence :math:`e^{-i \omega t}` is omitted,
Expand All @@ -52,21 +52,21 @@ The first one is the inevitable spatial singularity at the source. The
second is the grid refinement requirements in order to capture the rapid
change of the primary field [1]_. In order to mitigate these issues,
PETGEM used a secondary field approach where the total electric
field :math:`\mathbf E` is obtained as:
field :math:`\mathbf E` is obtained as

.. math::
\mathbf E &= \mathbf E_{p} + \mathbf E_{s} \\
\tilde{\sigma} &= \tilde{\sigma_{s}} + \Delta \tilde{\sigma}
\mathbf E &= \mathbf E_{p} + \mathbf E_{s}, \\
\tilde{\sigma} &= \tilde{\sigma_{s}} + \Delta \tilde{\sigma},
:label: total_electric_field
where subscripts :math:`p` and :math:`s` represent a primary field and
secondary field respectively. For a general layered Earth model,
:math:`\mathbf E_{p}` can be computed semi-analytically by using Hankel
transform filters. Based on this decomposition and following the work by
[3]_ the equation system to solve :math:`\mathbf E_{s}` is:
[3]_ the equation system to solve :math:`\mathbf E_{s}` is

.. math::
\nabla \times \nabla \times \mathbf E_{s} + i \omega \mu \tilde{\sigma} \mathbf E_{s} = -i \omega \mu \Delta \sigma \mathbf E_{p}
\nabla \times \nabla \times \mathbf E_{s} + i \omega \mu \tilde{\sigma} \mathbf E_{s} = -i \omega \mu \Delta \sigma \mathbf E_{p},
:label: electric_field_weak_form1
where the electrical conductivity :math:`\sigma` is a function of position
Expand All @@ -81,60 +81,60 @@ based on the skin depth of the electric field [4]_.
Edge finite element formulation
-------------------------------
For the computation of :math:`\mathbf E_{s}`, PETGEM implemented the
Nédélec EFEM which uses vector basis functions defined on the edges of
the corresponding elements. Its basis functions are divergence-free but
high-order Nédélec EFEM which uses vector basis functions defined on the edges,
faces and volume of the corresponding elements. Its basis functions are divergence-free but
not curl-free [2]_. Thus, EFEM naturally ensures tangential continuity and
allows normal discontinuity of :math:`\mathbf E_{s}` at material interfaces.
PETGEM used unstructured tetrahedral meshes because of their ability to
represent complex geological structures such as bathymetry or reservoirs as
well as the local refinement capability in order to improve the solution
accuracy. `Figure 4.1`_ shows the tetrahedral Nédélec elements (lowest order)
accuracy. `Figure 4.1`_ shows the tetrahedral Nédélec elements (first-, second-, and third-order)
together with their node and edge indexing.

.. _Figure 4.1:
.. figure:: _static/figures/edgeElement.jpg
:scale: 12%
:alt: Tetrahedral Nédélec edge element with node/edge indexing.
:scale: 25%
:alt: Tetrahedral Nédélec edge element with node/edge/face indexing.
:align: center

Figure 4.1. Tetrahedral Nédélec edge element with node/edge indexing.
Figure 4.1. Reference tetrahedral element showing numeration of DOFs for each order :math:`p`. For :math:`p=1` there are 6 DOFs (1 per edge). For :math:`p=2` there are 20 DOFs (2 per edge and 2 per face). Finally, for :math:`p=3` there are 45 DOFs (3 per edge, 6 per face and 3 per element's volume).

In PETGEM workflow, the tangential component of the secondary electric
field is assigned to the edges in the mesh. Therefore, all components of the
electric field at a point :math:`\mathbf x` located inside a tetrahedral
element :math:`e` can be obtained as follows:
element :math:`e` can be obtained as follows

.. math::
\mathbf E^{e}(\mathbf x) = \sum_{i=1}^{6} \mathbf N^{e}_{i}(\mathbf x) E^{e}_{i}
\mathbf E^{e}(\mathbf x) = \sum_{i=1}^{ndofs} \mathbf N^{e}_{i}(\mathbf x) E^{e}_{i},
:label: field_components_edge_element
where :math:`\mathbf N^{e}_{i}` are the vector basis functions associated
to each edge :math:`i` and :math:`E^{e}_{i}` their degrees of freedom.
Considering the node and edge indexing in `Figure 4.1`_, the vector basis
functions can be expressed as follows:
where :math:`\mathbf N^{e}_{i}` are the vector basis functions and :math:`E^{e}_{i}` their degrees of freedom.
For instance, considering the node and edge indexing in `Figure 4.1`_, the vector basis
functions for :math:`p=1` (first-order) can be expressed as follows

.. math::
\mathbf N^{e}_{i} &= (\lambda^{e}_{i1} \nabla \lambda^{e}_{i2} - \lambda^{e}_{i2} \nabla \lambda^{e}_{i1}) \ell^{e}_{i}
\mathbf N^{e}_{i} &= (\lambda^{e}_{i1} \nabla \lambda^{e}_{i2} - \lambda^{e}_{i2} \nabla \lambda^{e}_{i1}) \ell^{e}_{i},
:label: nedelec_basis
where subscripts :math:`i1` and :math:`i2` are the first and second nodes
linked to the :math:`i`-th edge, :math:`\lambda^{e}_{i}` are the linear
nodal basis functions, and :math:`\ell^{e}_{i}` is the length of the
:math:`i`-th edge of the element :math:`e`.
:math:`i`-th edge of the element :math:`e`. A systematic approach for obtaining
mixed-order curl-conforming basis functions may be seen in [4]_, [5]_.

By substituting equation :eq:`field_components_edge_element` into
:eq:`electric_field_weak_form1`, and using Galerkin's approach, the weak
form of the original differential equation becomes:
form of the original differential equation becomes

.. math::
Q_{i} = \int_{\Omega} \mathbf N_{i} \cdot [ \nabla \times \nabla \times \mathbf E_{s} -i \omega \mu \tilde{\sigma} \mathbf E_{s} + i \omega \mu \Delta \tilde{\sigma} \mathbf E_{p} ] dV
Q_{i} = \int_{\Omega} \mathbf N_{i} \cdot [ \nabla \times \nabla \times \mathbf E_{s} -i \omega \mu \tilde{\sigma} \mathbf E_{s} + i \omega \mu \Delta \tilde{\sigma} \mathbf E_{p} ] dV,
:label: electric_field_weak_form2
The compact discretized form of :eq:`electric_field_weak_form2` is
obtained after applying the Green's theorem:
obtained after applying the Green's theorem

.. math::
[K^{e}_{jk} + i \omega \tilde{\sigma}_{e} M^{e}_{jk}] \cdot \{ E_{sk} \} = - i \omega \mu \Delta \tilde{\sigma}_{e} R^{e}_k
[K^{e}_{jk} + i \omega \tilde{\sigma}_{e} M^{e}_{jk}] \cdot \{ E_{sk} \} = - i \omega \mu \Delta \tilde{\sigma}_{e} R^{e}_k,
:label: system_eq_edge_electric
where :math:`K^{e}` and :math:`M^{e}` are the elemental stiffness
Expand All @@ -146,3 +146,5 @@ side which requires numerical integration.
.. [2] Jin, J. (2002). The Finite Element Method in Electromagnetics. Wiley, New York, second edn.
.. [3] Newman, G.A. and Alumbaugh, D.L. (2002). Three-dimensional induction logging problems, Part 2: A finite difference solution. Geophysics, 67(2), 484–491.
.. [4] Puzyrev, V., Koldan, J., de la Puente, J., Houzeaux, G., Vázquez, M. and Cela, J.M. (2013). A parallel finite-element method for three-dimensional controlled-source electromagnetic forward modelling. Geophysical Journal International, ggt027.
.. [5] Garcia-Castillo, L.E., Salazar-Palma, M., 2000. Second-order Nédélec tetrahedral element for computational electromagnetics. International Journal of Numerical Modelling: Electronic Networks, Devices and Fields (John Wiley & Sons, Inc.) 13, 261–287.
.. [6] Garcia-Castillo, L.E., Ruiz-Genovés, A.J., Gómez-Revuelto, I., Salazar-Palma, M., Sarkar, T.K., 2002. Third-order Nédélec curl-conforming finite element. IEEE Transactions on Magnetics 38, 2370–2372.
Empty file modified doc/source/Contact.rst 100644 → 100755
Empty file.
8 changes: 5 additions & 3 deletions doc/source/Download.rst 100644 → 100755
Expand Up @@ -184,7 +184,7 @@ Downloads
Scripts
*******

* PETGEM is available for download at the project website generously hosted by PyPi. Download `here <https://pypi.python.org/pypi/petgem/>`__.
* PETGEM is available for download at the project website generously hosted by PyPi and GitHub. Download `here <https://pypi.python.org/pypi/petgem/>`__ or `here <https://github.com/ocastilloreyes/petgem>`__.

* ``kernel.py``: python script that manages the PETGEM work-flow. Download :download:`here <_downloads/kernel.py>`.

Expand All @@ -201,6 +201,8 @@ Examples

* Example 1: Dataset for Pre-processing stage within PETGEM. Download :download:`here <_downloads/Example1.zip>`.

* Example 2: Dataset for the Canonical model of an off-shore hydrocarbon reservoir. Download :download:`here <_downloads/Example2.zip>`.
* Example 2: Dataset for the Canonical model of an off-shore hydrocarbon reservoir (Nédélec elements of first-order). Download :download:`here <_downloads/Example2.zip>`.

* Example 3: Dataset for the use of PETSc solvers. Download :download:`here <_downloads/Example3.zip>`.
* Example 3: Dataset for the Canonical model of an off-shore hydrocarbon reservoir (Nédélec elements of second-order). Download :download:`here <_downloads/Example3.zip>`.

* Example 4: Dataset for the use of PETSc solvers. Download :download:`here <_downloads/Example4.zip>`.
11 changes: 6 additions & 5 deletions doc/source/Features.rst 100644 → 100755
Expand Up @@ -3,7 +3,7 @@
Features
===============

PETGEM use a code structure for the Nédélec FE algorithm that emphasizes
PETGEM use a code structure for the high-order Nédélec FE algorithm that emphasizes
good parallel scalability, which is crucial in the multi-core era.
Furthermore, it’s modularity should simplify the process of reaching the
best possible performance in terms of percentage of the peak amount of
Expand All @@ -30,15 +30,15 @@ A more detailed explanation is the following:

* Independent of problem formulation, numerical solution, and data storage: The kernel provides the independent abstractions for modeling, numerical methods, data storage and analysis.

* Parallel processing support: Based on distributed-memory parallelism (`petsc4py <https://pypi.python.org/pypi/petsc4py>`__) and static load balancing.
* Parallel processing support: Based on distributed-memory parallelism (`petsc4py <https://pypi.python.org/pypi/petsc4py>`__) and static load balancing. Further, GPUs architectures are supported.

* Confidence and performance monitoring: Based on an simple and automatic profiling module.

* Efficient solvers & preconditioners: Direct as well as iterative solvers and preconditioners are supported through `petsc4py <https://pypi.python.org/pypi/petsc4py>`__ package. As result, `PETSc <http://www.mcs.anl.gov/petsc/>`__ and `MUMPs <http://mumps.enseeiht.fr/>`__ libraries are supported.

* Interface to mesh generator: Simple and light library to export nodal finite element meshes to edge elements data structures. Current version support `Gmsh <http://gmsh.info/>`__ meshes.

* Edge FEM library: Edge-based discretisations, vector basis functions, their geometry description, and generalized integration rules provides a generic support for implementing edge-based solution algorithms.
* Edge FEM library: High-order Edge-based discretisations, vector basis functions, their geometry description, and generalized integration rules provides a generic support for implementing edge-based solution algorithms.

* Linear systems library: Support to Compressed Row Storage (CSR) format for sparse matrices and their easy and efficient parallel assembly on distributed-memory platforms.

Expand Down Expand Up @@ -80,8 +80,9 @@ Target architecture
--------------------
The HPC goal of the PETGEM involves using cutting-edge architectures.
To that goal, the code is implemented in current state-of-the-art
platforms such as Intel Xeon Platinum, Intel Haswell and Intel Xeon Phi processors, which offer
high performance, flexibility, and power efficiency. Nevertheless,
platforms such as Intel Xeon Platinum processors from the Skylake generation,
Intel Haswell and Intel Xeon Phi processors, which offer
high-performance, flexibility, and power efficiency. Nevertheless,
PETGEM support older architectures such as SandyBridge, for the sake
of usability and to be able to compare performance.

Expand Down
23 changes: 16 additions & 7 deletions doc/source/Installation.rst 100644 → 100755
Expand Up @@ -31,19 +31,19 @@ Install PETGEM

* Following commands may require root privileges

* Download `PETSc <https://www.mcs.anl.gov/petsc/>`__ (PETSc 3.7 and 3.8 have been tested)
* Download `PETSc <https://www.mcs.anl.gov/petsc/>`__ (PETSc 3.7, 3.8 and 3.9 have been tested)

* Uncompress the `PETSc <https://www.mcs.anl.gov/petsc/>`__ archive (in this example, using PETSc 3.8.3):
* Uncompress the `PETSc <https://www.mcs.anl.gov/petsc/>`__ archive (in this example, using PETSc 3.9.3):

.. code-block:: bash
$ tar zxvf petsc-3.8.3.tar.gz
$ tar zxvf petsc-3.9.3.tar.gz
* Configure and build `PETSc <https://www.mcs.anl.gov/petsc/>`__. The configuration options depend on the calculations you want to perform (complex- or real-valued) as well as your compiler/MPI/Blas/Lapack setup. For PETGEM executions, **PETSC MUST BE BUILD FOR COMPLEX-VALUED NUMBERS**. In order to avoid incompatibilities between PETSC, petsc4py and PETGEM, we highly recommend the following configuration lines. Please, visit PETSc website for advanced configuration options. If you have a clean environment (not working MPI/Blas/Lapack), then run:
* Configure and build `PETSc <https://www.mcs.anl.gov/petsc/>`__. The configuration options depend on the calculations you want to perform (complex- or real-valued) as well as your compiler/MPI/Blas/Lapack setup. For PETGEM executions, **PETSC MUST BE BUILD FOR COMPLEX-VALUED NUMBERS**. In order to avoid incompatibilities between PETSC, petsc4py and PETGEM, we highly recommend the following configuration lines. Please, visit `PETSc <https://www.mcs.anl.gov/petsc/>`__ website for advanced configuration options. If you have a clean environment (not working MPI/Blas/Lapack), then run:

.. code-block:: bash
$ cd petsc-3.8.3
$ cd petsc-3.9.3
$ export PETSC_DIR=$PWD
$ export PETSC_ARCH=arch-linux2-c-debug
Expand All @@ -59,6 +59,14 @@ Install PETGEM
$ --download-mumps --download-scalapack --download-parmetis --download-metis --download-ptscotch --download-cmake
* Further, to activate GPUs support, please add following options to previous configure line:

.. code-block:: bash
$ --with-cuda=1 --with_cuda_dir=PATH
where ``PATH`` is the directory of your CUDA libraries.

* Then, build and install `PETSc <https://www.mcs.anl.gov/petsc/>`__:

.. code-block:: bash
Expand Down Expand Up @@ -91,7 +99,8 @@ Downloading and building PETGEM
-------------------------------

The PETGEM package is available for download at
`Python Package Index (PyPI) <https://pypi.python.org/pypi/petgem/>`__
`Python Package Index (PyPI) <https://pypi.python.org/pypi/petgem/>`__, at
`GitHub <https://github.com/ocastilloreyes/petgem>`__,
and the :ref:`Download` section of this project website.

* Configure and install `PETSc <https://www.mcs.anl.gov/petsc/>`__ (see :ref:`Install` section)
Expand All @@ -111,7 +120,7 @@ and the :ref:`Download` section of this project website.
$ tar zxvf petgem-1.1.tar.gz
$ cd petgem-1.1
* After unpacking the release tarball, the distribution is ready for building. Some environment configuration is needed to inform the location `PETSc <https://www.mcs.anl.gov/petsc/>`__. As in :ref:`Install` section, you can set the environment variables ``PETSC_DIR`` and ``PETSC_ARCH`` indicating where you have built/installed `PETSc <https://www.mcs.anl.gov/petsc/>`__:
* After unpacking the release tarball, the distribution is ready for building. Some environment configuration is needed to inform the `PETSc <https://www.mcs.anl.gov/petsc/>`__ location. As in :ref:`Install` section, you can set the environment variables ``PETSC_DIR`` and ``PETSC_ARCH`` indicating where you have built/installed `PETSc <https://www.mcs.anl.gov/petsc/>`__:

.. code-block:: bash
Expand Down

0 comments on commit c98dcd2

Please sign in to comment.