optimize the CPML code (which is currently very slow) and use weights in Scotch decomposition for C-PML elements in the code

[mpbl]

use weights in Scotch decomposition for C-PML elements in the code;
because C-PML elements will be more expensive because they will compute more terms and will solve more equations (for the memory
variables / convolution terms) thus we should assign a higher weight to them when we call Scotch

the "elmnts_load" array up in decompose_mesh/part_decompose_mesh.f90 (around line 1432)
currently does not take into account C-PML elements weights. The matching array in the code is
called "CPML_type" and is defined as follows: 1 = face, 2 = edge, 3 = corner.

that is similar to what Daniel does for acoustic elements in the current version of SPECFEM3D: he uses a weight of 1 for acoustic
elements and 3 (or something like that) for elastic elements, which use a 3D vector instead of a scalar.

In principle the weighting factors can be computed analytically by counting the exact number of additional multiplications and
additions performed by C-PML elements

In PML corners this additional factor can be multiplied by 2 (for instance in the slice of PML that has X and Y damping) or by 3
for corners that combine an X, a Y and a Z PML.

easy to do once we have an expression for the weighting factors

[komatits]

already done by Zhinan Xie @xiezhinan I think (?).

Dimitri, June 2015: no, not done yet.

[komatits]

Fixing this would be easy (measuring the CPU time spent in each kind of elements with a CPU timer, and then using that as a weight when calling SCOTCH); but it is not done yet I think, unless Zhinan has done it?

[komatits]

James Smith @Jas11 and Dmitry Borisov @dborissov are currently doing it.

[komatits]

See also the comment at the end of #625

[komatits]

Vadim Monteiller @vmont, James Smith @Jas11 and Dmitry Borisov @dborissov are going to finish that jointly (as well as optimizing the CPML code, which is currently slow).

[komatits]

From Zhinan Xie @xiezhinan :

Computational cost with or without memory variables in an elastic element and PML element:

We have counted the computational work and memory variable in elastic element and PML element in both specfem2d and specfem3d.
They are as follows:

1.SPECFEM2D

Elastic element

Computational amount involved (NGLLX = 5)：1450

Memory variable ：0

Elastic PMLelement

Computational amount involved (NGLLX = 5)：3775 + 1450=5225

Memory variable：8NxNz+4NxNz=300

Viscoelastic element

Computational amount involved (NGLLX = 5)：1650+3N_SLS25

Memory variable：3*25

Viscoelastic PML element

Computational amount involved (NGLLX = 5)：5425 +2875*N_SLS

Memory variable：(12+6*N_SLS)

2.SPECFEM3D

Elastic element

Computational amount involved (NGLLX = 5)：74NxNyNz+24 NxNxNy*Nz

Memory variable： 0

Elastic PML element

Computational amount involved (NGLLX = 5)：(557+48Nx)NxNy*Nz

Memory variable：39NxNy*Nz

Viscoelastic element

Computational amount involved(NGLLX = 5)：

111NxNyNz+24NxNyNzNx+60N_SLSNxNyNz

Memory variable：6NxNyNzN_SLS

Viscoelastic PML element

Computational amount involved(NGLLX = 5)：

522NxNyNz+48NxNxNxNy+444N_SLSNxNx*Ny

Memory variable：29NxNxNy+9N_SLSNxNxNy

Best regards,

Zhinan

[komatits]

Note also that the PML source code itself could probably be optimized a lot (reducing the size of the arrays used, avoiding "if" statements in loops, precomputing things, factorizing things, vectorizing loops etc). This should of course be done before computing and assigning some PML weights to account for the load as explained above, since doing such optimizations will reduce the load and thus change the relative weights.