README

// The LANL computer code number is
// CLAMR -- LA-CC-11-094

////////////////
// GIT Commands
////////////////

To checkout the code

% git clone git@github.com:losalamos/CLAMR.git

////////////////////////
// Useful bash commands
////////////////////////

// logistics
locate [file]	// find file in directory
history		// see history of bash commands
source [file]	// read this file (in a shell form)

// code checking
valgrind	// check memory errors
// To run valgrind with libtool built code
libtool --mode=execute valgrind --tool=memcheck --leak-check=full --show-reachable=yes --track-origins=yes --leak-resolution=high --num-callers=20 --log-file=vgdump clamr_cpuonly -n 16 -i 1 -t 1

ddd		// debugger
totalview (there is a module on yellowrail and turing)

// code compiling
// We have switched to cmake so the configure step is 
cmake .                         // in-tree build
cmake <path-to-src>             // out-of-tree build
cmake -DCMAKE_BUILD_TYPE=debug <path-to-src>
cmake -DCMAKE_BUILD_TYPE=release <path-to-src>

cmake -DGRAPHICS_TYPE=None   <path-to-src>  // Turn off real-time graphics
cmake -DGRAPHICS_TYPE=OpenGL <path-to-src>  // Default
cmake -DGRAPHICS_TYPE=MPE    <path-to-src>  // Alternate real-time graphics

cmake -DPRECISION_TYPE=full_precision     // full double precision
cmake -DPRECISION_TYPE=mixed_precision    // intermediates double, arrays single
cmake -DPRECISION_TYPE=minimum_precision  // all single precision

// To build:
make
// or for a specific executable
make clamr_cpuonly

// To run the unit tests
make check
// or for a specific package unit tests
make MallocPlus_check

// To build the documentation
make doc
// or for a specific package documentation
make MallocPlus_doc

// To run on a Darwin MIC-NODE:
ssh darwin
salloc -p knc-mic
ssh [NODE]
// where [NODE] is $SLURM_NODELIST
// Load modules for build
module load compilers/intel/15.0.0
module load mpi/intelmpi-4.1.3.045-mic
module load cmake/2.8.11.1

/// These can be setup in the .login for the Intel MICs on darwin with
module load cmake/2.8.11.1    compilers/intel/15.0.0    mpi/intelmpi-4.1.3.045-mic
//The following source command can also be put in the .login with the first version for
// csh and the second for sh. When these are needed and what they do is still being studied
source /projects/opt/intel/compilers/composer_xe_2015.0.090/bin/compilervars.csh intel64
source /projects/opt/intel/compilers/composer_xe_2015.0.090/bin/compilervars.sh intel64

// For building on the host, change the module load to drop the -mic on the mpi
 module load cmake/2.8.11.1    compilers/intel/15.0.0    mpi/intelmpi-4.1.3.045

// Build CLAMR on MIC:
cd CLAMR
cmake -DMIC_NATIVE=yes
make
// To execute:
ssh [NODE]-mic0

// Execute clamr_openmponly
export OMP_NUM_THREADS=120
In the CLAMR directory
cd CLAMR
./clamr_openmponly

For Cielito (Cray XE6) architecture

// Note that CLAMR has some problems with OpenMPI, but works at least with this version
// of mpich
   module swap PrgEnv-pgi PrgEnv-gnu
   module load cray-mpich2/6.3.1
   module load cmake/2.8.9

cmake -DCMAKE_BUILD_TYPE=release -DCMAKE_CXX_COMPILER=`which CC` -DCMAKE_C_COMPILER=`which cc` .
    or the equivalent (graphics and precision types below are currently the defaults)
cmake -DCMAKE_BUILD_TYPE=release -DGRAPHICS_TYPE=None -DPRECISION_TYPE=full_precision -DCMAKE_CXX_COMPILER=`which CC` -DCMAKE_C_COMPILER=`which cc` .

// To build
   make
// There will be complaints about shared libraries. Go to each directory where it complains and
// edit the CMakeLists.txt and change "SHARED" to "STATIC"
   vi genmalloc/CMakeLists.txt
   vi graphics/CMakeLists.txt
   vi memstats/CMakeLists.txt
   vi mesh/hsfc/CMakeLists.txt
   vi mesh/kdtree/CMakeLists.txt
   vi mesh/zorder/CMakeLists.txt
   vi s7/CMakeLists.txt
   vi timer/CMakeLists.txt

// To run
   aprun -n 128 -N 16 ../clamr_mpionly -n 128 -t 1000

// Watch-out for memory leaks with open-mpi 1.8.5 on GPU/MPI versions

// Set MPEHOME to location of MPE install or MPE_INCLUDE_DIR and MPE_LIBRARIES to 
// the include path and the location of the MPE libraries

//
// need to edit configure.ac (b/c using autoconf package)
//
make				// builds the executable
make install			// copy the executable to destination
make clean			// restores to the pre-make step
make distclean			// restores to the pre-configure step
make installcheck		// not always there
make dist			// builds new distro into a new tar file
//
// need to edit Makefile.am (b/c using automake package)
//

/////////////////////
// Environment Setup
/////////////////////

// Update .bashrc by setting editor and loading modules
export EDITOR=vim
module load cuda
module load openmpi-gcc/1.4.1-64


/////////////////////
// Darwin
/////////////////////
To build on Darwin, The OpenCL libraries needed to build are only 
available on back-end nodes.  To get a node do:

Get a list of hardware partitions
% sinfo 

For a generic tesla. The Kepler can be requested with the "K20" partition
% salloc -p tesla -N 1
% ssh $SLURM_NODELIST

The following modules should be loaded. This line can also be added
to the .login file to make it automatic
% module load cmake/2.8.10.2 mpi/openmpi-1.6.4-gcc-4.4.7

You must use a special version of GCC for CUDA that is not the default
compiler on Darwin.  To use the correct compiler do:

% export PATH=/home/opt/gcc/4.4.6/bin:$PATH

Totalview on darwin
% module load totalview/8.11.0-2
% totalview <executable> -a <program command line>

Example
% totalview mpirun -a -n 2 ./clamr_mpionly -n 512 -l 2 -i 100 -t 10000

////////////////
// Moonlight
////////////////

%ssh -Y ml-fey

Get a backend node. The -nn is the number of nodes and the -np is the number
of processors per node
%llogin -nn 1 -np 8

Load modules
% module load gcc/4.7.2 openmpi/1.6.3  opencl/1.1

Add the following to the .login and you can select the gcc or intel compiler with
source ~/.login intel (for intel) or source ~/.login (for gcc)

========== .login file ============
if ("$1" =~ "intel") then
   module load intel/13.0.1
else
   module load gcc/4.7.2
endif

module load openmpi/1.6.3 opencl/1.1
===================================

********************* graphics and checkpoint restart ****************************

Just a little information on the movie graphics and the checkpoint/restart

The image and movie capabilities require the python imaging library (PIL or pillow
on Mac OSx) and mplayer (mencode comes with mplayer), respectively. An alternate
graphics output is done using the MagickWand library in many different formats.

For MAC:
brew install Homebrew/Python/pillow
brew install mplayer

For the MagickWand graphics library in ImageMagick 
brew install imagemagick

************************************* clamr *************************************
This is all the code I adjusted in CLAMR. Below are notes on the new flags I added.

-R: Restart CLAMR from a specified backup file, so you will need to pass the
    backup filename along with the path (EX: -R "checkpoint_output/backup00100.crx")
-b: The number of rollback files to maintain. Disk files are maintained with the 
    specified number of symlinks. In-memory images are limited to the specified
    number of rollback images
-c: Backup CLAMR states to disk at the specified cycle intervals. They will be
    put in the checkpoint_output directory
-C: Backup CLAMR states in-memory disk at the specified cycle intervals
-g: This generates the graphics files at the specified cycle intervals that are passed
    in by this flag. The graphics data files .data & .lin will be dumped in the directory
    where the executable is running. 
-G: graphics file type. Possible values include none, data, bmp, gif, jpeg, mpeg, pdf,
    png, and svg. Example -g 100 -G png will dump a graph#.png file in the graphics_output
    directory every 100 cycles.
-U: Upper bound for what is the allowable mass percentage difference. This number can
    be in integer, decimal, or scientific notation (EX: 1.23e-13).  


************************************* scripts ************************************
This is the code for generating the images and movies. The generate_image.py, 
generate_movie.sh, and FreeSansBold.ttf should be placed just outside of the graphics_output
directory. They will go into this directory to generate all image and movie data. The 
movies that are generated are avi files, but they can be converted to any format using
on-line video converters. I did this, because the best quality for the movies is 
generated by first making a avi file. The on-line movie converter I used was at 
video.online-convert.com. Mplayer can be used in Linux to just play the movie in an 
X-Terminal if you pass it the avi file.

FreeSansBold.tff: This is just the text file used by the python script to create the
    lettering for number of iterations and time steps in the images. This file, 
    generate_image.py, and generate_movie.sh are all needed to create the movies. They
    all just need to be in the same directory as the grahpics_info file that contains
    the graphics information from the simulation. If you just want to create the images
    and not the movie, then only this file and generate_image.py are needed in the 
    same directory as the graphics_output directory.

generate_image.py: This is python script that is used to generate the images from the
    CLAMR simulation. The image data needs to be a directory called graphics_output. The
    python script will go into this directory and generate the png files as well as an
    text file called image list, which is used to generate a movie. You can pass the y
    flag to this script if you want to have grid lines in the images as well. All image
    data is saved in the .data files and all gridlines data is saved in the .lin files.
    The FreeSansBold.ttf is needed to create the text in each image for the Iteration
    number and Time Step values.

generate_movie.sh: This script uses the python code above to first generate the images,
    and then generate a movie out the images. This script has a three flags that can be
    used. They are explained below. The movie that is generated is dumped in the current
    directory with the label test.avi
    -y: Defines whether you want gridlines in the images for the movie generated
        (False by default)
    -f: How many FPS you want the movie to be (6 by default)
    -s: What size do you want the images in the movie to be (800x800 by default).

So hopefully this all makes sense. I might have left something out. If you have any problems or
issues, just email me at bwa@g.clemson.edu. I will respond ASAP.