Skip to content

SPM support scripts for CBS researchers in NCF

Notifications You must be signed in to change notification settings

harvardinformatics/CBS_SPM

 
 

Repository files navigation

Scripts for Managing fMRI Data

#+ search mode org blah

Goals

This document describes the suite of scripts written to take a SPM batch file created for one subject and generate identical batch files for many subjects. These batch files are then submitted to the cluster via bsub or sbatch, eliminating the need for the GUI, and speeding up the process of analyzing (or reanalyzing) many subjects at once. In order to use the scripts, create your desired batch (preproc, or level1) in the SPM interface and use the ‘save as script’ option when saving the batch. This creates a file called batchname_job.m, which will serve as your template for the scripts below.

Prerequisites

System requirements: All scripts require that you be on the cluster, and were tested using Matlab 2012a with spm version 8.4667, and python version 2.7.3. Use with older versions of matlab is unsupported, and is known to fail using Matlab 2007. In order to access the scripts, you should be using the default environment as specified on the FAQ site (http://cbs.fas.harvard.edu/science/core-facilities/neuroimaging/information-investigators/faq). Please also see the FAQ for detailed information on how to select your matlab and SPM versions.

Directory structure: The scripts rely on the following directory structure which will automatically be created for you via the first script, getSubjectSPM.py.

|-mystudydir   
|---120418_mysubject
|------RAW
|---------dicom files...
|------analysisdir
|---------paradigms
|------------run001,run002,...
|---------------cond1.txt, cond2.txt...
|---------spm files from first level analysis...
|------batch
|---------spm batches...
|------preproc
|---------converted dicoms, preprocessed data...
|------output_files
|---------files containing error messages from scripts

Summary

The following scripts have been written and should be included in your default path if you are using the one listed on the FAQ site

Script NameDetails
getSubjectSPM.pyPulls data using cbsget and creates a directory structure
genPreprocBatches.pyGenerates preprocessing batches from template batch and executes batches
genL1Batches.pyGenerates level 1 batches from template and executes them

getSubjectSPM.py

getSubjectSPM.py is a script that pulls data from the CBSCentral repository (or a directory), converts it to SPM format (img or nii), and creates a directory structure that will allow the use of the other scripts described here.

It takes the following inputs in any order. For help, type getSubjectSPM.py -h:

FlagStands forDescription
-hhelpshows usage message and exits
-ssubjectidthe name of the subject
-bboldrunsa list of run numbers for the BOLD scans, seperated by spaces
-tstructrunthe run number for the structural scan
-mfieldmap runsthe run numbers for the fieldmap, if present needs two runs
-pdestination paththe FULL path to the location to place the output, no relative path (i.e. no ../ or ~)
-ddicom paththe path to a single directory containing all of the unzipped dicoms (as opposed to using CBSCentral
-nuse niftiIf present, will use the single file NIFTI format (nii) insted of img/hdr
–analysis-diranalysis directorya list of directories for future analyses. Default is analysis. Useful if collected multiple experiments in one session.
–vmemvirtural memoryamount of memory requested, requred if using sbatch, default is 1024MB
–timetimeamount of time script will need, required if using sbatch, default is 2 hours
-run-withwhat to execute the script with, choices are bsub, sbatch, or dry. dry means just output the call to getsubject.m, default is sbatch

For example:

getSubjectSPM.py -s 120418_spmtest -b 6 10 11 -t 3 -m 6 7 -p /ncf/mylab/myspace/myexp/ -n --analysis-dir analysis_univariate analysis_mvpa --run-with bsub

For getting dicom’s locally as opposed to CBSCentral:

getSubjectSPM.py -s 120418_spmtest -b 6 10 11 -t 3 -m 6 7 -p /ncf/mylab/myspace/myexp/ -d /ncf/mylab/myspace/myexp/dicoms/subj1

This creates the following directory tree:

|-myexp   
|----120418_spmtest
|-------RAW
|-------analysis
|----------paradigms
|-------------run001,run002,...
|-------batch
|-------preproc
|-------output_files

Within the RAW directory is a tarball (subjectid.tar.gz) containing the DICOMs in a compressed format. In the preproc directory will be the SPM converted files either nii or .img and .hdr.

The files have also been renamed. Because they are already in the subject directory, they have been stripped of their subjectid, and are renamed as follows below. Importantly, the functional runs will be named in the order you specify them after the -b flag. Thus, the first bold number specified wil be run001. This is useful if you collect data for multiple experiments in a scan session, and the order of experiments is varied across subjects. For instance, if you run a session with two studies, with three runs each, sometimes with expA first, some subjects with expB first. For ease of analysis later, you want the first three runs after renaming to always be expA, and runs 4-6 to be expB. You can do this by chaning the order of the bolds for the different subjects. -b 5 6 7 8 9 10 for those with expA first, and -b 8 9 10 5 6 7 with expB first.

File nameDescription
f-run001-006.imgImage 6 of the first BOLD run
s-struct.imgThe structural image for the subject
s-fieldmap-mag-01.imgThe magnitude of the fieldmap (if provided)
s-fieldmap_phase.imgThe phase of the fieldmap

Errors

If there is a problem with the script prior to it being submited to the cluster, the output will go to the screen (standard out) for debugging. This is true for things like the subject directory already exisiting.

If there is a problem prior to submitting the Arcget call, the error messages will be placed in the the study directory mystudy, with the name errors_getsubj_subjid followed by the date and time (to the min).

For example: errors_subjid_150101_subj1_2015_07_06_10h_23m

The output from the running of the Arcget command (that comes via the bsub/sbatch output) will be stored in mystudy, with the name output_getsubj_subjid followed by the date and time (to the min). This is where errors thrown by Arcget will show up, such as if the subject ID doesn’t exist.

For example: output_subjid_150101_subj1_2015_06_20_11h_41m

genPreprocBatches.py

The goal of this script is to take a batch file created to perform preprocessing on a single subject and use it to analyze many subjects. This is done by saving your batch via the 'save as script' command in SPM. This creates a batchname_job.m file, which will serve as your template batch. This batch will be applied to all of the subjects provided, which can include the original subject that was used to create the template. This script has been tested with fieldmap, slice time correction, motion correction, indirect spatial normalization, and smoothing. If you use any additional steps, you should check that the generated batches are correct by comparing the ones created to the original. Essenetially all it does is replace the listed path with the path to your new subject.

FlagStands forDescription
-hhelpprovides usage message and then exits
-ttemplate batchthe full path to, and name of the template batch created in the SPM GUI via a “save batch as script” command, that ends in _job.m
-ppaththe path to the directory that contains all of your subjects
-ssubjida subjid to create and execute the batch on, can be a list separated by spaces
-fsubject filea file containing your subjectids, with each ID on its own line, which can be used instead of -s flag
–vmemvirtural memoryamount of memory requested, requred if using sbatch, default is 1024MB
–timetimeamount of time script will need, required if using sbatch, default is 2 hours
–run-withwhat to execute the script with, choices are bsub, sbatch, or dry. dry means just make the new batch but don’t run it, default is sbatch

For example:

genPreprocBatches -t /ncf/mylab/myspace/myexp/subject1/batch/preproc_job.m -p /ncf/mylab/myspace/myexp/ -s subject1 subject2 --run-with bsub 

This will create a batch file for each subject provided, and save it in subjid/batches. It will then submit the created batch. You can check that your submitted jobs are running via the bjobs/sacct commands (see FAQ for instructions).

Errors

If there is a problem with converting the template batch for each subject, the error messages will be placed in the the study directory mystudy, with the name errors_preproc followed by the date and time (to the min).

For example: errors_preproc2012_07_06_10h_23m

The output from the running of the batch (that comes via the bsub output) will be stored in subjid/output_files, with the name output_preproc followed by the date and time (to the min). This is where errors thrown by matlab or SPM will show up.

For example: output_preproc2012_06_20_11h_41m

genL1Batches.py

The goal of this script is to take a batch file created to perform first level analysis on a single subject and use it to analyze many subjects. This is done by saving your batch via the ‘save as script’ command in SPM. This creates a batchname_job.m file, which will serve as your template batch. This batch will be applied to all of the subjects provided, which can include the original subject that was used to create the template. To run this script, you need to have your paradigm files constructed.

Creating batch

The names you use for your conditions will need to be the names of the text files containing your stimulus onset values (see below), so don’t put spaces in the name.

  1. When you make your contrasts in Contrast Manager, you can use either the T- and F-contrasts,

or the T-contrast (cond/sess based) options. The cond/sess method is preferred, as it is harder to make errors.

Running script

FlagStands forDescription
-hhelpprovides usage message and then exits
-ttemplate batchthe full path to, and name of the template batch created in the SPM GUI via a “save batch as script” command, that ends in _job.m
-ppaththe path to the directory that contains all of your subjects
-ssubjida subjid to create and execute the batch on, can be a list separated by spaces
-fsubject filea file containing your subjectids, with each ID on its own line, which can be used instead of -s
–vmemvirtural memoryamount of memory requested, requred if using sbatch, default is 1024MB
–timetimeamount of time script will need, required if using sbatch, default is 2 hours
–run-withwhat to execute the script with, choices are bsub, sbatch, or dry. dry means just make the new batch but don’t run it, default is sbatch

For example:

genL1Batches -t /ncf/mylab/myspace/myexp/subject1/batch/L1_job.m -p /ncf/mylab/myspace/myexp/ -s subject1 subject2

This will create a batch file for each subject provided, and save it in subjid/batches. It will then submit the created batch. You can check that your submitted jobs are running via the bjobs command (see FAQ for instructions).

Stimulus onset files

Within the analysis directory is a paradigms directory, with a directory for each run, run001. For the first level analysis, each condition should have it’s own onset text file, with each row being a single onset time. The name of the file should be the name given to each condition within the SPM batch, followed by the .txt extension, cond1.txt. Therefore, if you have 3 runs, you will end up with three text files for cond1. They will all be called cond1.txt, but placed in each run directory run001, run002, and run003. If your stimulus is presented 4 times per run, than each of those files will have 4 rows, with each row having the time in seconds (or TRs, depending on what you specify in your batch) when your stimulus was presented. These can be made up in matlab, or any text editor.

Errors

If there is a problem with converting the template batch for each subject, the error messages will be placed in the the study directory mystudy, with the name errors_L1 followed by the date and time (to the min).

For example: errors_L12012_07_06_10h_23m

The output from the running of the batch (that comes via the bsub/sbatch output) will be stored in subjid/output_files, with the name output_L1 followed by the date and time (to the min). This is where errors thrown by matlab or SPM will show up.

For example: output_L12012_06_20_11h_41m

Acknowledgments

These scripts were written by Alex Storer, Caitlin Carey, Stephanie McMains and Tim O’Keefe with additional assistance from David Dodell-Feder.

About

SPM support scripts for CBS researchers in NCF

Resources

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Python 63.5%
  • MATLAB 36.5%