Test Cases ucvm2mesh_mpi_layer

ucvm2mesh

ucvm2mesh_mpi configuration files

There are two inter-related configuration files used in the ucvm2mesh-mpi software. One file is the ucvm2mesh.conf, the other is the job submission script, that runs ucvm2mesh on a cluster. Examples of these two script are below:

A key constraint: the ucvm2mesh configuration file specifies the partitioning of grid among processor (as px, py, pz). The number of processes that are requested in the mpi job submission needs to be the product of these three values (e.g. example below (2 * 2 * 5) = 20). If the mpi job submission script requests more or less processes (aka tasks) than this number, the system exits with a configuration error.

example ucvm2mesh.conf

-bash-4.2$ more ucvm2mesh.conf
# List of CVMs to query
ucvmlist=cvmsi

# UCVM conf file
ucvmconf=/auto/scec-00/maechlin/ucvmc/conf/ucvm.conf

# Gridding cell centered or vertex (CENTER, VERTEX)
gridtype=CENTER

# Spacing of cells
spacing=2000.0

# Projection
proj=+proj=utm +datum=WGS84 +zone=11
rot=-40.0
x0=-122.3
y0=34.7835
z0=0.0

# Number of cells along each dim
nx=384
ny=248
nz=25

# Partitioning of grid among processors (request px*py*pz processes in mpi submit)
px=2
py=2
pz=5

# Vs/Vp minimum
vp_min=200
vs_min=200

# Mesh and grid files. Meshtype must be one of valid formats (IJK-12, IJK-20, IJK-32, SORD)
meshfile=/auto/scec-00/maechlin/ucvmc_mpi_testing/mesh_cvms5_sord_2000m_mpi.media
gridfile=/auto/scec-00/maechlin/ucvmc_mpi_testing/mesh_cvms5_sord_2000m_mpi.grid
meshtype=SORD

# Location of scratch dir
scratch=/staging/scratch

Then, the associate mpi job submission script must request the same number of processes. USC HPC, where we test ucvmc uses the slurm job manager. The following job manager script:

ucvm2mesh.slurm

-bash-4.2$ more *.slurm
#!/bin/bash
##SBATCH -p scec
#SBATCH --ntasks=20 #number of tasks with one per processor
#SBATCH -N 10
#SBATCH --mem 0 # Set to unlimited memory
#SBATCH --time=02:30:00
#SBATCH -o ucvm2mesh_mpi_large_%A.out
#SBATCH -e ucvm2mesh_mpi_large_%A.err
#SBATCH --export=NONE
#SBATCH --mail-user=maechlin@usc.edu
#SBATCH --mail-type=END

cd /auto/scec-00/maechlin/ucvmc_mpi_testing
srun -v --mpi=pmi2 /auto/scec-00/maechlin/ucvmc/bin/ucvm2mesh_mpi -f /auto/scec-00/maechlin/ucvmc_mpi_testing/ucvm2mesh.con

Installing and Running UCVM2mesh_mpi

1.	Building and running UCVMC v171:
We built and ran UCVMC v17.1 on USC hpc. We re-built it using openmpi libraries and used updated job submit scripts for slurm. 
- Examples of how to run ucvm2mesh_mpi are in my directory at 
- /home/scec-00/maechlin/ucvmc_mpi_testing/testcase6/
- There is a ucvm2mesh.conf and a makemesh.slurm which run successful for small meshes.

We tested ucvm2mesh and ucvm2mesh_mpi on small meshes and confirmed they produce the identical grid and media file results. So the rest of our testing was with ucvm2mesh_mpi.

2.	Output File Formats:

We looked into the output files and formats, and the explanation I made on our call was wrong. The numbers are not related to significant digits in the output files.

We determined that a standard ucvm2mesh_mpi run produces an output grid file, and a mesh file (also called a media file) 

The grid file contains the I j and lat lon values in the mesh. There is a utility called dumpgridp.py in the ucvmc scripts directory that will read the grid file and print the I,j values and the associated lat,lon values.

The .media file formats include IJK-12, IJK-20, IJK-32 and SORD. I believe all mesh file formats are fast x, y, and z (although this is an assumption based on the grid file format). The formats differ by the contents for each mesh point. The size of each point depends on the size of a structure, and the structures contains different information, shown below:

typedef struct mesh_ijk12_t {
  float vp;
  float vs;
  float rho;
} mesh_ijk12_t;

typedef struct mesh_ijk20_t {
  float vp;
  float vs;
  float rho;
  float qp;
  float qs;
} mesh_ijk20_t;

typedef struct mesh_ijk32_t {
  int i;
  int j;
  int k;
  float vp;
  float vs;
  float rho;
  float qp;
  float qs;
} mesh_ijk32_t;
The SORD output format, produces three files. Each file is apparently only one material property value, and has no ijk values, written like this:

case MESH_FORMAT_SORD:
for (i = 0; i < node_count; i++) {
      ptr1_sord[i].val = nodes[i].vp;
      ptr2_sord[i].val = nodes[i].vs;
      ptr3_sord[i].val = nodes[i].rho;
    }

For the mesh format ijk-32 includes qp, and qs values based on this formula /* Qp/Qs via Kim Olsen */
qs = 50.0 * (vs / 1000.0);
qp = 2.0 * qs;

Also, we noted that if a undefined file format is given (e.g. if you leave out the dash in a string like IJK-12) the code complains, but doesn’t exit, and it will run till it is killed by the wall clock timer.

3.	Impact of Vertex/Center:

My explanation on the call for vertex, center was wrong. Testing shows that Vertex setting sets the origin at exactly the given point, while Center setting moves the lat/lon point to the center of the cell and queries properties there. For example, given a origin at 34.00, -118,0, these two grid files are produced:

Center result:
0, 0 : -117.994639, 34.004553
1, 0 : -117.983811, 34.004640
2, 0 : -117.972984, 34.004726

Vertex  result:
0, 0 : -118.000000, 34.000000
1, 0 : -117.989173, 34.000088
2, 0 : -117.978346, 34.000174
4.	Orientation of x, y and z:

We setup some small scale meshes, with and without rotations, with a different number of points in each direction. Our tests indicate negative rotation is clockwise, which is consistent with the source code comments that say “ proj rotation angle in degrees, (+ is counter-clockwise)”

We setup a mesh with 1km spacing, Center mesh, and dimensions of x=40pt, y=20pts, z=10pts and tried different rotations:

no rotation end points:
0, 0 : -117.994639, 34.004553
39, 19 : -117.573488, 34.178591

negative 90 rotation end points:
0, 0 : -117.994534, 33.995535
39, 19 : -117.785601, 33.64531

postive 90 rotation end points:
0, 0 : -118.005467, 34.004465
39, 19 : -118.216216, 34.354259

This is one result that I have concerns about. As expected, the three end points form right angles, but they do not look east/west or north/south as expected. Possibly the projection is making these look like they are not east/west, but this is what we got.

5.	Additional Notes:

A few other things we learned in our testing.

-	If we tried to query for a point, and none of the velocity model had data for that point, ucvm2mesh-mpi exited with an error. For this reason, when querying cvms5 (aka cvm-s4.26 which has no background model) we needed to add a background model in our model list, to build our example meshes.

-	Ucvm2mesh_mpi produces a useful summary about the mesh it just produced when it completes. This summary includes the max, min vs, mp and the point where they are found.

With MinVS and MinVS to to 0 (updated ucvm2mesh.conf to set min vp and min vs to 0)
[0] Max Vp: 8530.947266 at
[0]	i,j,k : 237, 53, 14
[0] Max Vs: 5225.743652 at
[0]	i,j,k : 237, 54, 13
[0] Max Rho: 3174.164062 at
[0]	i,j,k : 119, 60, 15
[0] Min Vp: 283.636932 at
[0]	i,j,k : 204, 110, 0
[0] Min Vs: 138.720001 at
[0]	i,j,k : 194, 82, 0
[0] Min Rho: 1909.786255 at
[0]	i,j,k : 204, 110, 0
[0] Min Ratio: 1.414214 at
[0]	i,j,k : 125, 62, 0

-	When you specify a min Vs and min Vp, ucvm2mesh-mpi only checks the min Vs value. If it finds Vs lower than the min Vs, it sets both the Vs and Vp to the min values specified in the ucvm2mesh config file.

if (vs < vs_min)
{
	vs = vs_min;
	vp = vp_min;
}

-	There is a mesh_check utility in the distribution. It checks to make sure that each record in the file is of the correct size. Furthermore, it checks to make sure that each value is not NaN, infinity, or negative.

  printf("Usage: %s input format\n\n",arg);
  printf("input: path to the input file\n");
  printf("format: format of file, IJK-12, IJK-20, IJK-32\n\n");

We generated a mesh and ran mesh_check and it returned:
-bash-4.2$ ../ucvmc/bin/mesh_check mesh_cvmsi_ijk32_2000m_mpi.media IJK-32
Record size is 32 bytes
Opening input file mesh_cvmsi_ijk32_2000m_mpi.media
Checked 2380800 vals total