MQ

MQ: pythonTools/mq.py

Description

The MQ tool carries out running MABE many times for replication while optionally varying parameters in a controlled way. This allows you to gather statistics on outcomes by observing evolution many times (replication) while varying the experimental condition. MQ relies on you having made a file that specifies the conditions for replication and parameter variation. See below for details.

Examples

Assuming you are running these commands from a folder with the mabe executable and myconditions.sh file:

• See all the command options
  python pythonTools/mq.py -h
• Don't run anything. Create directories and files to see if it is correct.
  python pythonTools/mq.py -f myconditions.sh -n
• Run locally (this is what you want if you are doing this on your own computer)
  python pythonTools/mq.py -f myconditions.sh -l

Usage

usage: mq.py [-h] [-n] [-l] [-t] [-d] [-f FILE_NAME]                          
                                                                              
optional arguments:                                                           
  -h, --help            show this help message and exit                       
  -n, --runNo           if set, will do everything (i.e. create files and     
                        directories) but launch jobs, allows you to do a dry  
                        run - default : false(will run)                       
  -l, --runLocal        if set, will run jobs localy - default : false(no     
                        action)                                               
  -t, --runTest         if set, will run jobs localy with 1 rep and 2 updates 
                        set - default : false(no action)                      
  -d, --runHPCC         if set, will deploy jobs with qsub on HPCC - default :
                        false(no action)                                      
  -f FILE_NAME, --file FILE_NAME                                              
                        file which defines conditions - default:              
                        mq_conditions.txt                                     

Note: On the HPCC at MSU you can use the -d option (--runHPCC long form) instead of running locally to have MQ submit jobs to the supercomputer compute queue. This allows you to run hundreds of replicates and parameter variations quite quickly.

-f or --file

MQ by default looks for a conditions file called mq_conditions.txt in the current working directory so would run with simply a python pythonTools/mq.py -l assuming mq_conditions.txt is in the same folder as you are currently in. You can override this by telling MQ which conditions file to use with this flag. For example: python pythonTools/mq.py -f myconditions.sh -l

-l or --runLocal

MQ will run all MABE replicates in series directly on the current computer. This is the most commong way to run if you have only a personal computer such as a laptop or desktop.

-t or --runTest

MQ will try to force MABE to run with only a single replicate and for only a couple updates. The purpose is so you can quickly see if everything runs correctly, and generates the files you hope to see at the end of each replicate execution.

-n or --runNo

MQ will not run MABE at all, but instead generate the directories and files up until running MABE. Additionally, MQ will list for you the conditions which would be run, and the total number of jobs (instances of running MABE) that would be created. For example, 2 unique conditions and replicates of 1 through 4 would yield 8 total jobs.

-d or --runHPCC

Instead of running on your local computer, MQ will submit all jobs to the local supercomputer which is currently assumed to be Michigan State University's HPCC. If you are not running this command on a dev-node on the HPCC then this command will produce errors or have no effect.

Conditions File

MQ reads a conditions file (it looks for ./mq_conditions.txt by default). This file will look something like this:

# Lines starting with # are ignored                                                                                       

# REPS = [FIRST REP] [LAST REP]                                                                                                                                                
REPS = 101 110                                                                                                            
                                                                                                                          
# Settings to override the config files, but will not be varied                                                           
CONSTANT = GLOBAL-updates 10  WORLD-worldType Berry                                                                      
                                                                                                                          
# VAR = [SHORT NAME]    [REAL NAME]     [conditon1,condition2,etc.]                                                       
# Short name is used in this file, and also determines output directory names                                             
VAR = TSK       WORLD_BERRY-taskSwitchingCost                                                                             
VAR = RF1       WORLD_BERRY-rewardForFood1                                                                                
VAR = RF2       WORLD_BERRY-rewardForFood2                                                                                
                                                                                                                          
# Alternatively to VAR/EXCEPT, conditions can achieve a similar effect and still respect EXCEPT declarations              
#CONDITIONS = TSK=1.0,3.0  RF1="some message here"  RF2=2,[1,2,3,4]                                                       
#CONDITIONS = TSK=1.0      RF1=0.80                                                                                       
                                                                                                                          
# EXCEPT = [SHORT NAME]=[condition],[SHORT NAME]=[condition],etc.                                                         
# If all name/condition pairs are met for any EXCEPT, then that combination will not be run.                              
EXCEPT =  TSK=3.0  RF1=0.50                                                                                               
#EXCEPT = TSK=3.0  RF2=0.75                                                                                               
                                                                                                                          
# list of setting files (.cfg) which you want MABE to load with the -f option. files will be loaded in the order provided.
SETTINGS = settings.cfg,settings_organism.cfg,settings_world.cfg                                                          
                                                                                                                          
# list of files used by MABE (i.e. maps, images, etc.) and you can use rep replacement {{rep}} syntax                     
# these files will be copied to the working directory if necessary                                                        
#OTHERFILES = my_file.txt                                                                                                 
                                                                                                                          
# a population_loader.plf file can be created using the contents here                                                     
# be sure to set the -p GLOBAL-initPop population_loader.plf in CONSTANT definitions above                                
#PLF = MASTER = 'LOD_organisms.csv'                                                                                       
# or the following 2 lines with rep replacement works:                                                                    
#PLF = some_var = best 5 by ID from { 'LOD_organisms_{{rep}}.csv' }                                                       
#PLF = MASTER = collapse some_var                                                                                         
-----                                                                                                                     
                                                                                                                          
# JOBNAME will be appended before condition (C*) identifier - if "NONE" then job will have not JOBNAME                    
JOBNAME = NONE                                                                                                            
                                                                                                                          
# EXECUTABLE is the name of program to run, may include a path                                                            
EXECUTABLE = ./mabe                                                                                                       

Let's go through each important command in this syntax. Occurrance is how many times the command can appear in the file.

REPS

Occurrance: 1
Given a start number and end number, the difference between the two is how many times each unique condition will be run with unique random number start conditions. For solving zero-padding and sorting issues, we run, for instance, 30 jobs by starting at 11 and stopping at 40. For 300 jobs we start at 101 and stop at 400.
REPS 101 300

VAR

Occurrance: 0-Many
Parameters often have long names in MABE. When optionally varying parameters it's nice to use a short word to identify the parameter to which you're referring.
VAR = RF1 WORLD_BERRY-rewardForFood1 here, RF1 is the short word you can use later and will be used to name files and folders.

CONDITIONS

Occurrance: 0-Many
Each CONDITIONS line lets you specify several series of parameter variations which should all be combined together in every combination. For instance, CONDITIONS = TSK=1.0,3.0 RF1=mylabel RF2=2,4 will result in 4 separate conditions, each of which will run for however many replicates you specified with REPS. The conditions would be: TSK=1.0 RF1=mylabel RF2=2, TSK=3.0 RF1=mylabel RF2=2, TSK=1.0 RF1=mylabel RF2=4, TSK=3.0 RF1=mylabel RF2=4. The names of the variables come from the name you created earlier using the VAR command.

EXCEPT

Occurrance: 0-Many
Each EXCEPT line indicates a specific combination of parameters which should NOT be run. For example, EXCEPT = TSK=3.0 RF1=2 would not run the combination of conditions from the previous example: TSK=3.0 RF1=mylabel RF2=2.

SETTINGS

Occurrance: 0-1
Only useful for running MABE on a supercomputer. Comma separated list of files. This line allows you to pass settings files to MABE at runtime using the -f flag. These files should be passed if you have changed some of the parameters and want those changes applied to all experimental conditions and all replicates. Alternatively, you can use the CONSTANT syntax and include all non-default settings directly in the conditions file instead of relying on separate settings files.

CONSTANT

Occurrance: 0-Many
Each CONSTANT line lets you specify any number of parameters and their values, space-separated. You may spread out parameters among multiple CONSTANT lines. For example, CONSTANT = GLOBAL-updates 10 would set the number of updates to 10 for all replicates and all conditions. Find these names from the .cfg settings files you can generate by running the mabe executable with the -s generate settings files flag.

OTHERFILES

Occurrance: 0-1
Only useful for running MABE on a supercomputer. Comma separated list of files. If your experiment requires additional files, such as images, experiment-specific settings, or special binaries, you may specify them here and they will be copied to the same place where each replicate of MABE will be run. The {{rep}} keyword (with braces) if found on this line will be replaced with the replicate number. This lets you copy unique files based on replicate number. Support for more keywords is planned.

PLF

Occurrance: 0-Many
Each PLF line will be written directly to a .plf file before each replicate of MABE is run. This allows you to specify population loading commands from the conditions file on a per-replicate basis. The {{rep}} keyword (with braces) if found on this line will be replaced with the replicate number. This lets you load a unique population for each replicate. Support for more keywords is planned.