Pipeline options

Pipeline Options

Our pipeline actually contains MEG processing functionality, and this helps to explain each option.

Each of these steps is implemented by a shell script of the same name found in the MEG script repository

It's possible to run these scripts manually, like so:

$ cd /cluster/kuperberg/SemPrMM/MEG/scripts
$ ./preProc_setup.sh ya1;./preProc_reject.sh ya1;./preProc_avg.sh ya1;./preProc_cov ya1;

But using the pipeline gives you a few advantages:

• Automatically logs the output of each step to MEG/data/$sub/logs
• Process many subjects in parallel using the --ncpu N option, e.g:

    $ semprmm_pipeline.py --preProc_setup --preProc_reject ya1 ya2 ya3 ya4 --ncpu 4
    (--preProc_setup and then --preProc_reject runs for each subject, at the same time)

• When a step completes, you get a nice email with the log file as the message

###Order of operations

You can specify these options in any order on the commandline, but they are applied to the subject as they appear in this document.

--preProc_all

This is a nice convenience function that passes each preProc option to the subject.

    $ semprmm_pipeline.py --preProc_all ya1
    (is equal to)
    $ semprmm_pipeline.py --prProc_setup --preProc_reject --preProc_avg --preProc_cov ya1

--preProc_setup
---------------

*Before running this, create a textfile as `MEG/$sub/$sub_bad_chan.txt`. This file can be empty.*

Within MEG/data/$sub, this option does the following:


*   This option copies the raw fif files raw_backup/
*   The Baleen fif files are renamed to BaleenLP (runs 1-4) and BaleenHP (runs 5-8).
*   Events are extracted from the raw fif files into eve/
*   The fixTriggers.py script is called (this works on the .eve files)
*   The EEG channels are renamed to normal channel names (FP1, CPZ, etc.)
*   Using the $sub_bad_chan.txt file, the bad channels are marked.

--preProc_reject
----------------

This step is pretty complex...

First, the pipeline writes out rej/reject.m that looks like this:

    fif2rej('$sub_ATLLoc_raw.fif');
    fif2rej('$sub_MaskedMMRun1_raw.fif');
    fif2rej('$sub_MaskedMMRun2_raw.fif');
    fif2rej('$sub_BaleenLPRun1_raw.fif');
    fif2rej('$sub_BaleenLPRun2_raw.fif');
    ...
    end;

This file contains a call to fif2rej for each raw fif file.

Next, the preProc_reject.sh script is called, which does the brunt of the work. This script starts matlab and passes in this rej/reject.m file.  fif2rej is called for every fif. But what does `fif2rej` do?

First, it checks for `MEG/data/$sub/rej/rej_thr.txt`. If this file exists for a subject, then it loads the file and uses these thresholds for subsequent processing. If this files doesn't exist, it uses the default file at `MEG/scripts/rej_thr.txt`. It then does the following for the veog, heog, and eeg channels:

*   Load the fif header with fiff_setup_read_raw
*   Load the channels of interest, but not bad channels
*   Remove the mean across all channels
*   For veog and heog, do the following at each time point:
    *   Compute the mean for a 100 msec window before and after the time point
    *   If the difference between these means is greater the threshold, this sample is considered a rejection point.
*   For the eeg channels, do this:
    *   LPF at 40 Hz and HPF at .5 Hz.
    *   Apply a 100 uV threshold to the data.
    *   Points that exceed this threshold are considered artifacts.
    
All three methods are very fast. 

After this finishes, `rej2eve.py` called. This file looks at the rejected samples produced by `fif2rej` and the `Mod.eve` files and rewrites the events whose samples intersect with the rejected samples and writes these new eve files to `eve/*ModRej.eve`.

**Note that the epoch lengths which will be scanned for artifacts are hard-coded into this script, so you should make sure that they match with the epoch lengths which are defined in makeAveFiles.py for actual analysis.**

Finally, the `cmpEve.py` script is called. This compares the `Mod.eve` and `ModRej.eve` files and stores a small dictionary in `eve/info.txt` with the amount of "surviving" events in the ModRej.eve and total events in the `Mod.eve` file.

###If you run this manually

**Not Recommended**

You need to make a file at `MEG/data/$sub/rej/reject.m` that looks like it's description above.  This pipeline does this for you automatically.

After you run `reject.m` in matlab, you need to do this from the command line:

    $ python /cluster/kuperberg/MEG/scripts/rej2eve.py {Your subject}
    $ python /cluster/kuperberg/MEG/scripts/cmpEve.py {Your subject}

--preProc_avg
-------------

This writes out .ave files and computes the average fif files.  

For projection on and off, this is done:

*   For each paradigm, a LP filter of 20Hz is used
*   For ATLLoc, a .5 Hz HP filter is used
*   Each run is averaged.
*   The two MaskedMM runs are grand-averaged into $sub\_MaskedMM\_All-ave.fif
*   The four BaleenHP runs are grand-averaged in $sub\_BaleenHP\_All-ave.fif
*   The four BaleenLP runs are grand-averaged in $sub\_BaleenLP\_All-ave.fif
*   The two AXCPT runs are grand-averaged into $sub\_AXCPT\_All-ave.fif

--preProc_cov
-------------
This writes out .cov files and computes the covariance fif files.

A LP filter of 20Hz is applied to each paradigm.

--preAnat
---------

This does all the anatomical preparation for source localization--e.g. organizing FLASH scans, making BEMs. You must have run the recon before you run this. After this runs, you need to do two manual steps:

1. Check the surfaces in tkmedit to make sure they look normal:
   tkmedit subj## T1.mgz, then load surfaces

2. Do the manual coregistration in mne_analyze

--makeInv
---------

This makes the forward and inverse solutions. If you have not done the manual coregistration first, this will fail.

--makeSTC
---------

This is more properly part of the analysis, not part of preprocessing. It makes STC files that can be manipulated by Matlab scripts for further analysis. It is part of the pipeline because that makes it easy to create STCs for many subjects at once, but it is more customizable than the other pipeline scripts because you have lots of parameter options and STC files take awhile to create, so you will probably just want to create the ones that you need right now. Therefore before running this you need to go into the makeSTC.sh script and comment out/in the lines desired. 


--setup_bem
-----------

can be called independently to set up bem, but usually you would just call --preAnat, which as its first step runs setup_bem


--run_ica
---------

*OLD and DEPRECATED: DON'T USE ME*
This option starts a MATLAB job that computes the ICA for the given subject.  It uses EEGLAB's implementation of the Independent Component Analysis.

This option makes *_ica.mat files along with *.blinks, where the * stands for [subject]\_[studyrun].

See [ICA](/KuperbergLab/MEG_scripts/wiki/ICA)
*AGAIN DON'T USE ME*