Running twolevel_ants_dbm on Niagara (old)

These instructions are old, see optimized_antsMultivariateTemplateConstruction

The twolevel_ants_dbm pipeline is a longitudinal or cross sectional pipeline to produce unbiased deformation based morphometry outputs, suitable for voxel wise analysis. Full details of the pipeline is available at https://github.com/CobraLab/twolevel_ants_dbm

This document will only cover the specifics of running this pipeline on Niagara, exact configuration of the pipeline is left to the reader to decide on based on the documentation for the pipeline itself.

There are two main differences for the default pipeline script compared to special runs on Niagara:

1. The cluster integration is inappropriately configured to talk to Niagara. A modified version of antsMulivariateTemplateConstruction2.sh must be used, which integrates job submission via qbatch.
2. You must run your pipeline inside a screen or tmux session as the pipeline needs to monitor and submit jobs as steps are completed.

Special one time setup instructions

This pipeline uses python and requires a few specific modules, as such you need to install them for your individual user.

In order to use this pipeline on Niagara, run this one time:

> module load cobralab/2019b
> pip install --user tqdm pathos

Running

First, after logging into Niagara, please note the hostname of the machine you are on, you will need to remember it if you lose your connection and need to get back to the running pipeline.

# Example
> hostname
nia-login06.scinet.local

You will need to run your pipeline in a detachable terminal multiplexer, either screen or tmux

Run your mutliplexer:

> screen # or tmux!

Load appropriate modules (or add them to a run script, even better!):

> module load cobralab/2019b

Get a copy of the twolevel GitHub directory

> git clone https://github.com/CoBrALab/twolevel_ants_dbm.git

Finally, after determining the configuration of interest based on https://github.com/CobraLab/twolevel_ants_dbm, add to your command

--modelbuild-command ${QUARNTINE_PATH}/twolevel_dbm/niagara2-antsMulivariateTemplateConstruction2.sh --cluster-type=slurm

This points the pipeline at the custom template builder and tells it to use SLURM.

You should now have a bash script to run the pipeline:

#!/bin/bash

module load cobralab/2019b

export PYTHONUNBUFFERED=TRUE

twolevel_ants_dbm/twolevel_dbm.py \
    --modelbuild-command ${QUARANTINE_PATH}/twolevel-dbm/niagara2-antsMultivariateTemplateConstruction2.sh \
	--cluster-type=slurm [possibly other options] 1level input.csv 2>&1 | tee -a dbm_logfile.log

And run it:

> bash my-dbm-run-script.sh

Note: when creating the csv file to run the 2level, each row is a subject and each column is the timepoint, such that each cell is the nifti filename for that subject at that timepoint.