Running the Graple Program

Running Simulations using Graple

Running a simulation using HT-Condor, IPOP, and Python is a reasonably straightforward endeavor.

Prerequisites:

1. An x86 PC running Windows 7 or Windows Server 2008 R2.
2. IPOP VPN
3. HT-Condor
4. Python 2.7
5. GRAPLE Program

Notation:

Sim – a single GLM simulation. Job – an HTCondor job, consisting of one or more GLM simulations that are archived together and executed serially on an HTCondor node. Jobs include more than one simulation because using HTCondor to execute short-running simulations (e.g. seconds) is not quite effective.

Simulation format and layout

To leverage the functionality of Graple, you will need to organize the files and directories of your simulation in the following manner. Create a top level folder, e.g., “SimRoot”, which will contain all the files pertaining to your experiment. Running Graple from this location will automatically create the necessary directories -- Condor, GLM, Graple, Results, Scripts, Sims. These directories will contain the following files:

• Condor – the Condor scripts used to schedule the jobs. These scripts are automatically generated only when needed.
• GLM – the GLM executable and libraries for running the simulations. You are required to put the appropriate binaries for a fully functional setup here. This includes the GLM executable and its associated dynamically linked libraries (DLL).
• Graple – the core program file for job batching which optimizes workload and simplifies usage. This file is available for download here.
• Scripts – One or more R scripts used for post-processing the GLM output. These are your scripts that you can use to post process the GLM output.
• Sims – contains sub-directories of simulation folders. Each one is a self-contained simulation to be run and should be named Simxxx (where xxx is a monotonic increasing numerical suffix). Typically each Simxxx directory contains the NML and CSV files associated with each individual experiment.
• Results - The output from your experiments will be placed here.

There are also two additional files, SubmitGrapleBatch.py and ProcessGrapleBatchOutputs.py, in the SimRoot folder that further simply submitting and collecting results.

IMAGE Placeholder

Workflow

There are 3 phases of Graple’s workflow – Simulation Submission, Execution, and Results Collation.

###Simulation Submission This is the mode for Graple when you invoke SubmitGrapleBatch. A series of tar/bz2 compressed archives are created, where each one stores all information to execute a batch Job scheduled with HTCondor. Each archive contains the GLM, Scripts, and one or more Sims/Simxxx folders. HTOn the Condor execute node all files in the archive will be unpacked to a single unique directory and the simulation executed.

###Simulation Execution This is automatically invoked on the HTCondor execute node. For each simulation Graple will run GLM and place its output in a Results sub-directory. It will then invoke the top level R script with the Results directory path. The output of R should be placed in the same folder. Upon completion of all the simulaltions, Graple will archive the all the outputs that were run in the job. The Results archive will then be transferred to the host that initiated the job.

Results Collation

Wait until all your jobs are finished running -- this can be verified with condor_q (your queue should have no jobs). When your simulations are all complete, invoke ProcessGrapleBatchOutputs.py to retrieve the results. This script will unpack the results archives for all the jobs that have been completed. The results are copied into the SimsRoot/Results/Simxxx/results directory.

Usage - quick start guide

1. Download [Graple setup archive]https://github.com/GRAPLE/htcondor_glm/releases/download/v1.0-beta.2/Graple-v1_0-Beta2.zip)
2. Extract the contents of the archive to a location of your choice, eg., _c:_ .
3. From the SimRoot folder, run "python CreateWorkingFolders.py" (or double-click -- see below)
4. Copy the proper files into the GLM, Scripts & Sims folders.
5. From the SimRoot folder, run "python SubmitGrapleBatch.py" (or double-click -- see below)
6. Check for job completion using condor_q in a command-line terminal
7. From the SimRoot folder, run "python ProcessGrapleBatchOutputs.py" (or double-click -- see below)
   Note that you can also double click the py files from within windows explorer - but you need to make sure the default program to open .py files is Python. You can do this by right-clicking the file, click the "Change..." button in "opens with", and select Python