Cokurt Summit Dev Env

Ascent + Pele Workflow

The intended workflow for development is, at a high level, a two step process.

1. Run pele with a CPU version of ascent to save out data files (hdf5) using the ascent integration.

2. Use Ascent's replay utility to feed those hdf5 files through the GPU version of Ascent into the cokurt filter.

We can all use the hdf5 files that are produced from Pele+Ascent. That said, there may be reasons to re-run the simulation. For example, we may need to save out a different set of variables or fix any issues with export in Ascent.

Ascent Dependencies

I have installed the dependencies for both cuda and non-cuda versions of ascent in: /gpfs/alpine/world-shared/csc340/cokurt/

The idea is to build your own version of Ascent for development and running. For the cokurt code, you will need to build the cuda version of Ascent.

Building GPU version of Ascent for development

Cuda CMake file:

/gpfs/alpine/world-shared/csc340/cokurt/ascent_cuda/uberenv_libs/login3-linux-rhel7-power9le-gcc@6.4.0-ascent.cmake

OpenMP CMake file:

/gpfs/alpine/world-shared/csc340/cokurt/ascent_openmp/uberenv_libs/login1-linux-rhel7-power9le-gcc@6.4.0-ascent.cmake

Building Ascent

Modules

ml cuda/10.1.168 
ml gcc/6.4.0
ml cmake/3.14.2

Clone and Build

git clone --recursive https://github.com/Alpine-DAV/ascent.git
cd ascent
git checkout cokurt
mkdir build
cd build
cmake -C /gpfs/alpine/world-shared/csc340/cokurt/ascent_cuda/uberenv_libs/login3-linux-rhel7-power9le-gcc@6.4.0-ascent.cmake ../src
make

Data

I put a single time step in a public directory. I am re-running the simulation with 1 layer of ghost, and this time I filled the ghosts with valid data (mf.FillBoundary()). This data should be a good start for testing that you can execute the learn filter.

/gpfs/alpine/world-shared/csc340/cokurt/pele_970.tar.gz

Inside the tar file is a .root file and a directory containing all of the simulation domains.

Using Replay

The replay utility is located here: build/utilities/replay/ Information about how to use it can be found in the documentation.

Using the learn filter

In order to call the learn filter, use the following actions:

-
  action: "add_extracts"
  extracts:
    e1:
      type: "learn"
      params:
        fields:
         - rho.Y(NC12H26)
         - rho.Y(HO2)
         - rho.Y(OH)
         - rho.Y(CH2O)
         - rho.Y(C12H24)
         - rho.Y(H2O)
         - temp

Running on GPU Nodes

To get an interactive session:

bsub -W 30 -nnodes 1 -P CSC340 -Is /bin/bash

-W is the number of winutes. -P is the project number to charge the time against. You can use one node, if you want to take MPI out of the equation, but for speed, I would recommend using 2-4 nodes, which means you are using 12-24 GPUs.

Exalearn Directory

/gpfs/alpine/cmb103/world-shared/exalearn/genten_tree