prep_data2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Prepare BOLD5000 data for input into a deep learning model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook takes the BOLD5000 dataset and prepares it for use in a deep learning model. Since the dataset is the direct output of fmriprep, we still need to perform a few steps before it is ready to train on.\n",
    "\n",
    "First, we must regress out nuisance signals. In fMRI analysis, many nuisance signals are gathering from the processing pipeline and are linearly regressed or detrended from the whole brain timeseries. Some commonly used nuisance signals include motion parameters (rigid body motion involves three translation, three rotation parameters), average CSF signal, average white matter signal, and all their derivatives. The regression of the global signal - or the mean signal across all brain voxels - is highly contended in the fMRI field. Many consider it to be physiological noise, others consider it to contain important neural information. For our purposes, we will not regress out global signal. Our goal is to decode visual stimuli features from fMRI timeseries of visual areas of the brain. A recent study on the global signal showed that the visual cortex contains much of variance in the global signal topography [1]. For this reason, we will only regress out six motion parameters, two biological, and their derivatives. We can pull these signals from one of the fmriprep files.\n",
    "\n",
    "Second, we must extract the timeseries data of brain regions that are involved in visual processing. We will use the visual regions defined in the HCP MMP 1.0 atlas. We will take these binary masks and mutiply them to each subject's fMRI data to extract the information we need.\n",
    "\n",
    "Finally, we need to label each part of the timeseries data with the image it corresponds to and package the resulting data into neat X and Y matrices.\n",
    "\n",
    "[1] Li, J., Bolt, T., Bzdok, D., Nomi, J. S., Yeo, B. T. T., Spreng, R. N., & Uddin, L. Q. (2019). Topography and behavioral relevance of the global signal in the human brain. Scientific Reports, 9(1), 1–10. https://doi.org/10.1038/s41598-019-50750-8"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import os\n",
    "import sys\n",
    "from glob import glob\n",
    "import pandas as pd\n",
    "import subprocess\n",
    "from scipy.io import savemat\n",
    "\n",
    "\"\"\"\n",
    "Takes a subject dataframe loaded from the regressors.tsv file and extracts FD.\n",
    "Performs motion scrubbing by marking all volumes with FD greater than threshold with 1.\n",
    "Additionally marks one volume before and two after. Converts to a sparse matrix where\n",
    "each column has one frame that is censored.\n",
    "\"\"\"\n",
    "def get_censored_frames(subj_df, threshold):\n",
    "    # Threshold should be bounded between 0 and 1\n",
    "    if threshold < 0 or threshold > 1:\n",
    "        raise ValueError('Threshold should be bounded between 0 and 1.')\n",
    "    # Extract FD column\n",
    "    fd = subj_df['FramewiseDisplacement'].values\n",
    "    fd = np.nan_to_num(fd)\n",
    "    # Create censor vector\n",
    "    censor = [0 if m <= threshold else 1 for m in fd]\n",
    "    # Censor one back, two forward\n",
    "    censor_fixed = np.zeros_like(censor)\n",
    "    for ind,c in enumerate(censor):\n",
    "        if c == 1:\n",
    "            try:\n",
    "                censor_fixed[ind-1:ind+3] = 1\n",
    "            except IndexError:\n",
    "                censor_fixed[ind-1:] = 1\n",
    "    # Convert to sparse matrix\n",
    "    censor_feat = np.zeros((censor_fixed.shape[0], np.count_nonzero(censor_fixed)))\n",
    "    col = 0\n",
    "    for ind,c in enumerate(censor_fixed):\n",
    "        if c == 1:\n",
    "            censor_feat[ind,col] = 1\n",
    "            col +=1\n",
    "\n",
    "    return censor_feat, censor_fixed\n",
    "\n",
    "\"\"\"\n",
    "Takes a subject dataframe loaded from the regressors.tsv file and extracts relevant regressors (list)\n",
    "\"\"\"\n",
    "def get_regressors(subj_df, regressors):\n",
    "    # Should be of dim TRs x # regressors\n",
    "    regress_mat = np.array([subj_df[regressor].values for regressor in regressors]).T\n",
    "    # Calculate derivatives manually\n",
    "    deriv = np.diff(regress_mat,axis=0)\n",
    "    deriv = np.insert(deriv, 0, regress_mat[0], axis = 0)\n",
    "    final = np.hstack((regress_mat,deriv))\n",
    "    return final\n",
    "\n",
    "\"\"\"\n",
    "Returns subject directories from fmriprep directory\n",
    "\"\"\"\n",
    "def get_subj_dirs(fmriprep_dir):\n",
    "    subj_dirs = [f for f in os.listdir(fmri_dir) if os.path.isdir(os.path.join(fmri_dir, f)) and 'sub' in f]\n",
    "    return subj_dirs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Registration and nuisance signal regression"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The subjects' fMRI data are in their native scanner space. It will be difficult to extract the same regions if there are not all aligned to the same standard space. We will do this using FLIRT to perform an affine transformation to the MNI standard space before we regress."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "PARAMETERS - change these if you wish\n",
    "\n",
    "fmri_dir: where the fmriprep output data lives, should contain subject folders\n",
    "nuisance_dir: where the confounds_regressors.tsv files are located, should contain subject folders\n",
    "regressors_dir: an output directory to hold the nuisance regressors text files\n",
    "preproc_dir: an output directory for the fully processed subject data\n",
    "\n",
    "\"\"\"\n",
    "fd_threshold = 0.5 # Threshold of FD for censoring a frame\n",
    "# All the nuisance regressors we wish to remove. Do not include derivatives, these are calculated manually\n",
    "regressors = ['CSF', 'WhiteMatter','X','Y','Z','RotX','RotY','RotZ'] \n",
    "# Set directories\n",
    "fmri_dir, nuisance_dir, regressors_dir, preproc_dir, standard = ['dataset/ds001499-download/',\n",
    "                                                       'dataset/ds001499-download/derivatives/fmriprep/',\n",
    "                                                       'dataset/regressors/',\n",
    "                                                       'dataset/preprocessed/',\n",
    "                                                       '/usr/share/fsl/5.0/data/standard/MNI152_T1_2mm_brain.nii.gz']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing sub-CSI1\n",
      "\tSession 1 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 2 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 3 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 4 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 5 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 6 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 7 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 8 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 9 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 10 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 11 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 12 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 13 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 14 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 15 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 16 out of 16\n",
      "Processing sub-CSI2\n",
      "\tSession 1 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 2 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 3 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 4 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 5 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 6 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 7 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 8 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 9 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 10 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 11 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 12 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 13 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 14 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 15 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 16 out of 16\n",
      "Processing sub-CSI3\n",
      "\tSession 1 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 2 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 3 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 4 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 5 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 6 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 7 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 8 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 9 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 10 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 11 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 12 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 13 out of 16\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 14 out of 16\n",
      "\t\tRun 1 out of 9\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 15 out of 16\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 16 out of 16\n",
      "Processing sub-CSI4\n",
      "\tSession 1 out of 10\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 2 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 3 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 4 out of 10\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 5 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 6 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 7 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 8 out of 10\n",
      "\t\tRun 1 out of 9\n",
      "\t\tRun 2 out of 9\n",
      "\t\tRun 3 out of 9\n",
      "\t\tRun 4 out of 9\n",
      "\t\tRun 5 out of 9\n",
      "\t\tRun 6 out of 9\n",
      "\t\tRun 7 out of 9\n",
      "\t\tRun 8 out of 9\n",
      "\t\tRun 9 out of 9\n",
      "\tSession 9 out of 10\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 10 out of 10\n"
     ]
    }
   ],
   "source": [
    "# Get all subject directories\n",
    "subj_dirs = get_subj_dirs(fmri_dir)\n",
    "\n",
    "# Loop through each subjects and get regressors, perform scrubbing\n",
    "for subj in sorted(subj_dirs):\n",
    "    print('Processing %s' % subj)\n",
    "    # Absolute path of current subject\n",
    "    subj_dir_abs = os.path.join(fmri_dir, subj)\n",
    "    sess_dirs = sorted([f for f in os.listdir(subj_dir_abs) if os.path.isdir(os.path.join(subj_dir_abs, f)) and 'ses-' in f])\n",
    "    if not sess_dirs: # If there are not multiple sessions, then set to list of empty string to iterate only once in for loop\n",
    "        sess_dirs = ['']\n",
    "    for sessnum,sess in enumerate(sess_dirs):\n",
    "        print('\\tSession %d out of %d' % ((sessnum + 1), len(sess_dirs)))\n",
    "        # Absolute path of current session\n",
    "        sess_dir_abs = os.path.join(subj_dir_abs, sess)\n",
    "        conf_sess_dir_abs = os.path.join(nuisance_dir, subj, 'ses-' + str(sessnum+1).zfill(2))\n",
    "        bold_files = sorted(glob(sess_dir_abs + '/func/*task-5000scenes*bold.nii.gz'))\n",
    "        confound_files = sorted(glob(conf_sess_dir_abs + '/func/*task-5000scenes*confounds*.tsv'))\n",
    "        # For multiple runs\n",
    "        for runnum, (bold, confound) in enumerate(zip(bold_files, confound_files)):\n",
    "            print('\\t\\tRun %d out of %d' % ((runnum + 1), len(bold_files)))\n",
    "            df = pd.read_csv(confound, sep='\\t')\n",
    "            censor_mat, censor_frames = get_censored_frames(df, fd_threshold)\n",
    "            regress_mat = get_regressors(df, regressors)\n",
    "            nuisance_mat = np.hstack((censor_mat,regress_mat))\n",
    "            prefix = os.path.join(regressors_dir, subj + '_ses-' + str(sessnum+1).zfill(2) + '_run-' + str(runnum+1).zfill(2) + '_')\n",
    "            outfile = prefix + 'censored.txt'\n",
    "            np.savetxt(outfile, censor_frames)\n",
    "            outfile = prefix + 'nuisance_regressors.txt'\n",
    "            np.savetxt(outfile, nuisance_mat)\n",
    "            # Perform registration\n",
    "            bold_reg = bold[:-7] + '_registered.nii.gz'\n",
    "            outmat = os.path.join(sess_dir_abs,'func',subj + '_ses-' + str(sessnum+1).zfill(2) + '_run-' + str(runnum+1).zfill(2) + '_native2template.mat')\n",
    "            cmd = 'flirt -in ' + bold + ' -ref ' + standard + ' -omat ' + outmat\n",
    "            subprocess.call(cmd, shell = True)\n",
    "            cmd = 'flirt -in ' + bold + ' -ref ' + standard + ' -out ' + bold_reg + ' -init ' + outmat + ' -applyxfm -v'\n",
    "            subprocess.call(cmd, shell = True)\n",
    "            # Use AFNI to perform regression\n",
    "            outfile = outfile[:-3] + 'mat'\n",
    "            savemat(outfile, {'nuisance_regressors': regress_mat})\n",
    "            subprocess.call('python2 /home/ubuntu/abin/read_matlab_files.py -infiles ' + outfile + ' -prefix ' + prefix[:-1] + ' -overwrite', shell = True)\n",
    "            design = glob(prefix[:-1] + '*.1D')[0]\n",
    "            prefix = os.path.join(preproc_dir, subj + '_ses-' + str(sessnum+1).zfill(2) + '_run-' + str(runnum+1).zfill(2) + '_')\n",
    "            outfile = prefix + 'preproc.nii.gz'\n",
    "            subprocess.call('3dTproject -input ' + bold_reg + ' -prefix ' + outfile + ' -ort ' + design + ' -polort 2 -passband 0.009 0.1 -blur 6 -quiet -overwrite', shell = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### ROI masking and train data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that we have the fMRI data fully preprocessed, we will need to extract the ROI timeseries of all the visual region masks that BOLD5000 kindly provided, and match the correct labels from the events file. The result will be an X and Y for each ROI mask. The X will be of shape (samples, timepoints, features) and Y will be of shape (samples, classes). Since each stimuli is presented for about 1 seconds with 9 seconds in between each, we will take 10 second windows for each class label. With a TR of 2 seconds, each sample will have 5 TRs of all the voxels in that ROI for each corresponding class label. The datasets will be concatenated for all runs, sessions, and subjects. Thus, the resulting data will be saved in the output directory specified (data_dir), one X and Y for each ROI.\n",
    "\n",
    "Again, stick with Python 2 for the sake of the AFNI commands used. Also, be sure to specify the directories at the top of the following cell.\n",
    "\n",
    "This will take only a few hours to run."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "from collections import defaultdict\n",
    "import nibabel as nib\n",
    "\n",
    "\"\"\"\n",
    "DIRECTORIES - BE SURE TO SET THESE!!\n",
    "\n",
    "roi_dir: where the roi_masks are located, should contain subject folders\n",
    "preproc_dir: where the fully processed data is stored\n",
    "events_dir: where the event files are held, should contain subject folders.\n",
    "            This is probably the same folder as the original dataset/fmriprep folder.\n",
    "data_dir: the output of where you want the training data to be saved\n",
    "mask_dir: the output of where you want your resampled roi masks to be\n",
    "            \n",
    "\"\"\"\n",
    "preproc_dir, events_dir, data_dir, mask_dir = ['dataset/preprocessed/',\n",
    "                                            'dataset/ds001499-download/',\n",
    "                                            'dataset/traindata/',\n",
    "                                            'dataset/masks/']\n",
    "\n",
    "# Dicts holding training set and labels for each mask\n",
    "X = []\n",
    "Y = []\n",
    "\n",
    "# Holds subject, session, run information for each sample\n",
    "sample2image2session = pd.DataFrame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed file 1 out of 510\n",
      "Preprocessed file 2 out of 510\n",
      "Preprocessed file 3 out of 510\n",
      "Preprocessed file 4 out of 510\n",
      "Preprocessed file 5 out of 510\n",
      "Preprocessed file 6 out of 510\n",
      "Preprocessed file 7 out of 510\n",
      "Preprocessed file 8 out of 510\n",
      "Preprocessed file 9 out of 510\n",
      "Preprocessed file 10 out of 510\n",
      "Preprocessed file 11 out of 510\n",
      "Preprocessed file 12 out of 510\n",
      "Preprocessed file 13 out of 510\n",
      "Preprocessed file 14 out of 510\n",
      "Preprocessed file 15 out of 510\n",
      "Preprocessed file 16 out of 510\n",
      "Preprocessed file 17 out of 510\n",
      "Preprocessed file 18 out of 510\n",
      "Preprocessed file 19 out of 510\n",
      "Preprocessed file 20 out of 510\n",
      "Preprocessed file 21 out of 510\n",
      "Preprocessed file 22 out of 510\n",
      "Preprocessed file 23 out of 510\n",
      "Preprocessed file 24 out of 510\n",
      "Preprocessed file 25 out of 510\n",
      "Preprocessed file 26 out of 510\n",
      "Preprocessed file 27 out of 510\n",
      "Preprocessed file 28 out of 510\n",
      "Preprocessed file 29 out of 510\n",
      "Preprocessed file 30 out of 510\n",
      "Preprocessed file 31 out of 510\n",
      "Preprocessed file 32 out of 510\n",
      "Preprocessed file 33 out of 510\n",
      "Preprocessed file 34 out of 510\n",
      "Preprocessed file 35 out of 510\n",
      "Preprocessed file 36 out of 510\n",
      "Preprocessed file 37 out of 510\n",
      "Preprocessed file 38 out of 510\n",
      "Preprocessed file 39 out of 510\n",
      "Preprocessed file 40 out of 510\n",
      "Preprocessed file 41 out of 510\n",
      "Preprocessed file 42 out of 510\n",
      "Preprocessed file 43 out of 510\n",
      "Preprocessed file 44 out of 510\n",
      "Preprocessed file 45 out of 510\n",
      "Preprocessed file 46 out of 510\n",
      "Preprocessed file 47 out of 510\n",
      "Preprocessed file 48 out of 510\n",
      "Preprocessed file 49 out of 510\n",
      "Preprocessed file 50 out of 510\n",
      "Preprocessed file 51 out of 510\n",
      "Preprocessed file 52 out of 510\n",
      "Preprocessed file 53 out of 510\n",
      "Preprocessed file 54 out of 510\n",
      "Preprocessed file 55 out of 510\n",
      "Preprocessed file 56 out of 510\n",
      "Preprocessed file 57 out of 510\n",
      "Preprocessed file 58 out of 510\n",
      "Preprocessed file 59 out of 510\n",
      "Preprocessed file 60 out of 510\n",
      "Preprocessed file 61 out of 510\n",
      "Preprocessed file 62 out of 510\n",
      "Preprocessed file 63 out of 510\n",
      "Preprocessed file 64 out of 510\n",
      "Preprocessed file 65 out of 510\n",
      "Preprocessed file 66 out of 510\n",
      "Preprocessed file 67 out of 510\n",
      "Preprocessed file 68 out of 510\n",
      "Preprocessed file 69 out of 510\n",
      "Preprocessed file 70 out of 510\n",
      "Preprocessed file 71 out of 510\n",
      "Preprocessed file 72 out of 510\n",
      "Preprocessed file 73 out of 510\n",
      "Preprocessed file 74 out of 510\n",
      "Preprocessed file 75 out of 510\n",
      "Preprocessed file 76 out of 510\n",
      "Preprocessed file 77 out of 510\n",
      "Preprocessed file 78 out of 510\n",
      "Preprocessed file 79 out of 510\n",
      "Preprocessed file 80 out of 510\n",
      "Preprocessed file 81 out of 510\n",
      "Preprocessed file 82 out of 510\n",
      "Preprocessed file 83 out of 510\n",
      "Preprocessed file 84 out of 510\n",
      "Preprocessed file 85 out of 510\n",
      "Preprocessed file 86 out of 510\n",
      "Preprocessed file 87 out of 510\n",
      "Preprocessed file 88 out of 510\n",
      "Preprocessed file 89 out of 510\n",
      "Preprocessed file 90 out of 510\n",
      "Preprocessed file 91 out of 510\n",
      "Preprocessed file 92 out of 510\n",
      "Preprocessed file 93 out of 510\n",
      "Preprocessed file 94 out of 510\n",
      "Preprocessed file 95 out of 510\n",
      "Preprocessed file 96 out of 510\n",
      "Preprocessed file 97 out of 510\n",
      "Preprocessed file 98 out of 510\n",
      "Preprocessed file 99 out of 510\n",
      "Preprocessed file 100 out of 510\n",
      "Preprocessed file 101 out of 510\n",
      "Preprocessed file 102 out of 510\n",
      "Preprocessed file 103 out of 510\n",
      "Preprocessed file 104 out of 510\n",
      "Preprocessed file 105 out of 510\n",
      "Preprocessed file 106 out of 510\n",
      "Preprocessed file 107 out of 510\n",
      "Preprocessed file 108 out of 510\n",
      "Preprocessed file 109 out of 510\n",
      "Preprocessed file 110 out of 510\n",
      "Preprocessed file 111 out of 510\n",
      "Preprocessed file 112 out of 510\n",
      "Preprocessed file 113 out of 510\n",
      "Preprocessed file 114 out of 510\n",
      "Preprocessed file 115 out of 510\n",
      "Preprocessed file 116 out of 510\n",
      "Preprocessed file 117 out of 510\n",
      "Preprocessed file 118 out of 510\n",
      "Preprocessed file 119 out of 510\n",
      "Preprocessed file 120 out of 510\n",
      "Preprocessed file 121 out of 510\n",
      "Preprocessed file 122 out of 510\n",
      "Preprocessed file 123 out of 510\n",
      "Preprocessed file 124 out of 510\n",
      "Preprocessed file 125 out of 510\n",
      "Preprocessed file 126 out of 510\n",
      "Preprocessed file 127 out of 510\n",
      "Preprocessed file 128 out of 510\n",
      "Preprocessed file 129 out of 510\n",
      "Preprocessed file 130 out of 510\n",
      "Preprocessed file 131 out of 510\n",
      "Preprocessed file 132 out of 510\n",
      "Preprocessed file 133 out of 510\n",
      "Preprocessed file 134 out of 510\n",
      "Preprocessed file 135 out of 510\n",
      "Preprocessed file 136 out of 510\n",
      "Preprocessed file 137 out of 510\n",
      "Preprocessed file 138 out of 510\n",
      "Preprocessed file 139 out of 510\n",
      "Preprocessed file 140 out of 510\n",
      "Preprocessed file 141 out of 510\n",
      "Preprocessed file 142 out of 510\n",
      "Preprocessed file 143 out of 510\n",
      "Preprocessed file 144 out of 510\n",
      "Preprocessed file 145 out of 510\n",
      "Preprocessed file 146 out of 510\n",
      "Preprocessed file 147 out of 510\n",
      "Preprocessed file 148 out of 510\n",
      "Preprocessed file 149 out of 510\n",
      "Preprocessed file 150 out of 510\n",
      "Preprocessed file 151 out of 510\n",
      "Preprocessed file 152 out of 510\n",
      "Preprocessed file 153 out of 510\n",
      "Preprocessed file 154 out of 510\n",
      "Preprocessed file 155 out of 510\n",
      "Preprocessed file 156 out of 510\n",
      "Preprocessed file 157 out of 510\n",
      "Preprocessed file 158 out of 510\n",
      "Preprocessed file 159 out of 510\n",
      "Preprocessed file 160 out of 510\n",
      "Preprocessed file 161 out of 510\n",
      "Preprocessed file 162 out of 510\n",
      "Preprocessed file 163 out of 510\n",
      "Preprocessed file 164 out of 510\n",
      "Preprocessed file 165 out of 510\n",
      "Preprocessed file 166 out of 510\n",
      "Preprocessed file 167 out of 510\n",
      "Preprocessed file 168 out of 510\n",
      "Preprocessed file 169 out of 510\n",
      "Preprocessed file 170 out of 510\n",
      "Preprocessed file 171 out of 510\n",
      "Preprocessed file 172 out of 510\n",
      "Preprocessed file 173 out of 510\n",
      "Preprocessed file 174 out of 510\n",
      "Preprocessed file 175 out of 510\n",
      "Preprocessed file 176 out of 510\n",
      "Preprocessed file 177 out of 510\n",
      "Preprocessed file 178 out of 510\n",
      "Preprocessed file 179 out of 510\n",
      "Preprocessed file 180 out of 510\n",
      "Preprocessed file 181 out of 510\n",
      "Preprocessed file 182 out of 510\n",
      "Preprocessed file 183 out of 510\n",
      "Preprocessed file 184 out of 510\n",
      "Preprocessed file 185 out of 510\n",
      "Preprocessed file 186 out of 510\n",
      "Preprocessed file 187 out of 510\n",
      "Preprocessed file 188 out of 510\n",
      "Preprocessed file 189 out of 510\n",
      "Preprocessed file 190 out of 510\n",
      "Preprocessed file 191 out of 510\n",
      "Preprocessed file 192 out of 510\n",
      "Preprocessed file 193 out of 510\n",
      "Preprocessed file 194 out of 510\n",
      "Preprocessed file 195 out of 510\n",
      "Preprocessed file 196 out of 510\n",
      "Preprocessed file 197 out of 510\n",
      "Preprocessed file 198 out of 510\n",
      "Preprocessed file 199 out of 510\n",
      "Preprocessed file 200 out of 510\n",
      "Preprocessed file 201 out of 510\n",
      "Preprocessed file 202 out of 510\n",
      "Preprocessed file 203 out of 510\n",
      "Preprocessed file 204 out of 510\n",
      "Preprocessed file 205 out of 510\n",
      "Preprocessed file 206 out of 510\n",
      "Preprocessed file 207 out of 510\n",
      "Preprocessed file 208 out of 510\n",
      "Preprocessed file 209 out of 510\n",
      "Preprocessed file 210 out of 510\n",
      "Preprocessed file 211 out of 510\n",
      "Preprocessed file 212 out of 510\n",
      "Preprocessed file 213 out of 510\n",
      "Preprocessed file 214 out of 510\n",
      "Preprocessed file 215 out of 510\n",
      "Preprocessed file 216 out of 510\n",
      "Preprocessed file 217 out of 510\n",
      "Preprocessed file 218 out of 510\n",
      "Preprocessed file 219 out of 510\n",
      "Preprocessed file 220 out of 510\n",
      "Preprocessed file 221 out of 510\n",
      "Preprocessed file 222 out of 510\n",
      "Preprocessed file 223 out of 510\n",
      "Preprocessed file 224 out of 510\n",
      "Preprocessed file 225 out of 510\n",
      "Preprocessed file 226 out of 510\n",
      "Preprocessed file 227 out of 510\n",
      "Preprocessed file 228 out of 510\n",
      "Preprocessed file 229 out of 510\n",
      "Preprocessed file 230 out of 510\n",
      "Preprocessed file 231 out of 510\n",
      "Preprocessed file 232 out of 510\n",
      "Preprocessed file 233 out of 510\n",
      "Preprocessed file 234 out of 510\n",
      "Preprocessed file 235 out of 510\n",
      "Preprocessed file 236 out of 510\n",
      "Preprocessed file 237 out of 510\n",
      "Preprocessed file 238 out of 510\n",
      "Preprocessed file 239 out of 510\n",
      "Preprocessed file 240 out of 510\n",
      "Preprocessed file 241 out of 510\n",
      "Preprocessed file 242 out of 510\n",
      "Preprocessed file 243 out of 510\n",
      "Preprocessed file 244 out of 510\n",
      "Preprocessed file 245 out of 510\n",
      "Preprocessed file 246 out of 510\n",
      "Preprocessed file 247 out of 510\n",
      "Preprocessed file 248 out of 510\n",
      "Preprocessed file 249 out of 510\n",
      "Preprocessed file 250 out of 510\n",
      "Preprocessed file 251 out of 510\n",
      "Preprocessed file 252 out of 510\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed file 253 out of 510\n",
      "Preprocessed file 254 out of 510\n",
      "Preprocessed file 255 out of 510\n",
      "Preprocessed file 256 out of 510\n",
      "Preprocessed file 257 out of 510\n",
      "Preprocessed file 258 out of 510\n",
      "Preprocessed file 259 out of 510\n",
      "Preprocessed file 260 out of 510\n",
      "Preprocessed file 261 out of 510\n",
      "Preprocessed file 262 out of 510\n",
      "Preprocessed file 263 out of 510\n",
      "Preprocessed file 264 out of 510\n",
      "Preprocessed file 265 out of 510\n",
      "Preprocessed file 266 out of 510\n",
      "Preprocessed file 267 out of 510\n",
      "Preprocessed file 268 out of 510\n",
      "Preprocessed file 269 out of 510\n",
      "Preprocessed file 270 out of 510\n",
      "Preprocessed file 271 out of 510\n",
      "Preprocessed file 272 out of 510\n",
      "Preprocessed file 273 out of 510\n",
      "Preprocessed file 274 out of 510\n",
      "Preprocessed file 275 out of 510\n",
      "Preprocessed file 276 out of 510\n",
      "Preprocessed file 277 out of 510\n",
      "Preprocessed file 278 out of 510\n",
      "Preprocessed file 279 out of 510\n",
      "Preprocessed file 280 out of 510\n",
      "Preprocessed file 281 out of 510\n",
      "Preprocessed file 282 out of 510\n",
      "Preprocessed file 283 out of 510\n",
      "Preprocessed file 284 out of 510\n",
      "Preprocessed file 285 out of 510\n",
      "Preprocessed file 286 out of 510\n",
      "Preprocessed file 287 out of 510\n",
      "Preprocessed file 288 out of 510\n",
      "Preprocessed file 289 out of 510\n",
      "Preprocessed file 290 out of 510\n",
      "Preprocessed file 291 out of 510\n",
      "Preprocessed file 292 out of 510\n",
      "Preprocessed file 293 out of 510\n",
      "Preprocessed file 294 out of 510\n",
      "Preprocessed file 295 out of 510\n",
      "Preprocessed file 296 out of 510\n",
      "Preprocessed file 297 out of 510\n",
      "Preprocessed file 298 out of 510\n",
      "Preprocessed file 299 out of 510\n",
      "Preprocessed file 300 out of 510\n",
      "Preprocessed file 301 out of 510\n",
      "Preprocessed file 302 out of 510\n",
      "Preprocessed file 303 out of 510\n",
      "Preprocessed file 304 out of 510\n",
      "Preprocessed file 305 out of 510\n",
      "Preprocessed file 306 out of 510\n",
      "Preprocessed file 307 out of 510\n",
      "Preprocessed file 308 out of 510\n",
      "Preprocessed file 309 out of 510\n",
      "Preprocessed file 310 out of 510\n",
      "Preprocessed file 311 out of 510\n",
      "Preprocessed file 312 out of 510\n",
      "Preprocessed file 313 out of 510\n",
      "Preprocessed file 314 out of 510\n",
      "Preprocessed file 315 out of 510\n",
      "Preprocessed file 316 out of 510\n",
      "Preprocessed file 317 out of 510\n",
      "Preprocessed file 318 out of 510\n",
      "Preprocessed file 319 out of 510\n",
      "Preprocessed file 320 out of 510\n",
      "Preprocessed file 321 out of 510\n",
      "Preprocessed file 322 out of 510\n",
      "Preprocessed file 323 out of 510\n",
      "Preprocessed file 324 out of 510\n",
      "Preprocessed file 325 out of 510\n",
      "Preprocessed file 326 out of 510\n",
      "Preprocessed file 327 out of 510\n",
      "Preprocessed file 328 out of 510\n",
      "Preprocessed file 329 out of 510\n",
      "Preprocessed file 330 out of 510\n",
      "Preprocessed file 331 out of 510\n",
      "Preprocessed file 332 out of 510\n",
      "Preprocessed file 333 out of 510\n",
      "Preprocessed file 334 out of 510\n",
      "Preprocessed file 335 out of 510\n",
      "Preprocessed file 336 out of 510\n",
      "Preprocessed file 337 out of 510\n",
      "Preprocessed file 338 out of 510\n",
      "Preprocessed file 339 out of 510\n",
      "Preprocessed file 340 out of 510\n",
      "Preprocessed file 341 out of 510\n",
      "Preprocessed file 342 out of 510\n",
      "Preprocessed file 343 out of 510\n",
      "Preprocessed file 344 out of 510\n",
      "Preprocessed file 345 out of 510\n",
      "Preprocessed file 346 out of 510\n",
      "Preprocessed file 347 out of 510\n",
      "Preprocessed file 348 out of 510\n",
      "Preprocessed file 349 out of 510\n",
      "Preprocessed file 350 out of 510\n",
      "Preprocessed file 351 out of 510\n",
      "Preprocessed file 352 out of 510\n",
      "Preprocessed file 353 out of 510\n",
      "Preprocessed file 354 out of 510\n",
      "Preprocessed file 355 out of 510\n",
      "Preprocessed file 356 out of 510\n",
      "Preprocessed file 357 out of 510\n",
      "Preprocessed file 358 out of 510\n",
      "Preprocessed file 359 out of 510\n",
      "Preprocessed file 360 out of 510\n",
      "Preprocessed file 361 out of 510\n",
      "Preprocessed file 362 out of 510\n",
      "Preprocessed file 363 out of 510\n",
      "Preprocessed file 364 out of 510\n",
      "Preprocessed file 365 out of 510\n",
      "Preprocessed file 366 out of 510\n",
      "Preprocessed file 367 out of 510\n",
      "Preprocessed file 368 out of 510\n",
      "Preprocessed file 369 out of 510\n",
      "Preprocessed file 370 out of 510\n",
      "Preprocessed file 371 out of 510\n",
      "Preprocessed file 372 out of 510\n",
      "Preprocessed file 373 out of 510\n",
      "Preprocessed file 374 out of 510\n",
      "Preprocessed file 375 out of 510\n",
      "Preprocessed file 376 out of 510\n",
      "Preprocessed file 377 out of 510\n",
      "Preprocessed file 378 out of 510\n",
      "Preprocessed file 379 out of 510\n",
      "Preprocessed file 380 out of 510\n",
      "Preprocessed file 381 out of 510\n",
      "Preprocessed file 382 out of 510\n",
      "Preprocessed file 383 out of 510\n",
      "Preprocessed file 384 out of 510\n",
      "Preprocessed file 385 out of 510\n",
      "Preprocessed file 386 out of 510\n",
      "Preprocessed file 387 out of 510\n",
      "Preprocessed file 388 out of 510\n",
      "Preprocessed file 389 out of 510\n",
      "Preprocessed file 390 out of 510\n",
      "Preprocessed file 391 out of 510\n",
      "Preprocessed file 392 out of 510\n",
      "Preprocessed file 393 out of 510\n",
      "Preprocessed file 394 out of 510\n",
      "Preprocessed file 395 out of 510\n",
      "Preprocessed file 396 out of 510\n",
      "Preprocessed file 397 out of 510\n",
      "Preprocessed file 398 out of 510\n",
      "Preprocessed file 399 out of 510\n",
      "Preprocessed file 400 out of 510\n",
      "Preprocessed file 401 out of 510\n",
      "Preprocessed file 402 out of 510\n",
      "Preprocessed file 403 out of 510\n",
      "Preprocessed file 404 out of 510\n",
      "Preprocessed file 405 out of 510\n",
      "Preprocessed file 406 out of 510\n",
      "Preprocessed file 407 out of 510\n",
      "Preprocessed file 408 out of 510\n",
      "Preprocessed file 409 out of 510\n",
      "Preprocessed file 410 out of 510\n",
      "Preprocessed file 411 out of 510\n",
      "Preprocessed file 412 out of 510\n",
      "Preprocessed file 413 out of 510\n",
      "Preprocessed file 414 out of 510\n",
      "Preprocessed file 415 out of 510\n",
      "Preprocessed file 416 out of 510\n",
      "Preprocessed file 417 out of 510\n",
      "Preprocessed file 418 out of 510\n",
      "Preprocessed file 419 out of 510\n",
      "Preprocessed file 420 out of 510\n",
      "Preprocessed file 421 out of 510\n",
      "Preprocessed file 422 out of 510\n",
      "Preprocessed file 423 out of 510\n",
      "Preprocessed file 424 out of 510\n",
      "Preprocessed file 425 out of 510\n",
      "Preprocessed file 426 out of 510\n",
      "Preprocessed file 427 out of 510\n",
      "Preprocessed file 428 out of 510\n",
      "Preprocessed file 429 out of 510\n",
      "Preprocessed file 430 out of 510\n",
      "Preprocessed file 431 out of 510\n",
      "Preprocessed file 432 out of 510\n",
      "Preprocessed file 433 out of 510\n",
      "Preprocessed file 434 out of 510\n",
      "Preprocessed file 435 out of 510\n",
      "Preprocessed file 436 out of 510\n",
      "Preprocessed file 437 out of 510\n",
      "Preprocessed file 438 out of 510\n",
      "Preprocessed file 439 out of 510\n",
      "Preprocessed file 440 out of 510\n",
      "Preprocessed file 441 out of 510\n",
      "Preprocessed file 442 out of 510\n",
      "Preprocessed file 443 out of 510\n",
      "Preprocessed file 444 out of 510\n",
      "Preprocessed file 445 out of 510\n",
      "Preprocessed file 446 out of 510\n",
      "Preprocessed file 447 out of 510\n",
      "Preprocessed file 448 out of 510\n",
      "Preprocessed file 449 out of 510\n",
      "Preprocessed file 450 out of 510\n",
      "Preprocessed file 451 out of 510\n",
      "Preprocessed file 452 out of 510\n",
      "Preprocessed file 453 out of 510\n",
      "Preprocessed file 454 out of 510\n",
      "Preprocessed file 455 out of 510\n",
      "Preprocessed file 456 out of 510\n",
      "Preprocessed file 457 out of 510\n",
      "Preprocessed file 458 out of 510\n",
      "Preprocessed file 459 out of 510\n",
      "Preprocessed file 460 out of 510\n",
      "Preprocessed file 461 out of 510\n",
      "Preprocessed file 462 out of 510\n",
      "Preprocessed file 463 out of 510\n",
      "Preprocessed file 464 out of 510\n",
      "Preprocessed file 465 out of 510\n",
      "Preprocessed file 466 out of 510\n",
      "Preprocessed file 467 out of 510\n",
      "Preprocessed file 468 out of 510\n",
      "Preprocessed file 469 out of 510\n",
      "Preprocessed file 470 out of 510\n",
      "Preprocessed file 471 out of 510\n",
      "Preprocessed file 472 out of 510\n",
      "Preprocessed file 473 out of 510\n",
      "Preprocessed file 474 out of 510\n",
      "Preprocessed file 475 out of 510\n",
      "Preprocessed file 476 out of 510\n",
      "Preprocessed file 477 out of 510\n",
      "Preprocessed file 478 out of 510\n",
      "Preprocessed file 479 out of 510\n",
      "Preprocessed file 480 out of 510\n",
      "Preprocessed file 481 out of 510\n",
      "Preprocessed file 482 out of 510\n",
      "Preprocessed file 483 out of 510\n",
      "Preprocessed file 484 out of 510\n",
      "Preprocessed file 485 out of 510\n",
      "Preprocessed file 486 out of 510\n",
      "Preprocessed file 487 out of 510\n",
      "Preprocessed file 488 out of 510\n",
      "Preprocessed file 489 out of 510\n",
      "Preprocessed file 490 out of 510\n",
      "Preprocessed file 491 out of 510\n",
      "Preprocessed file 492 out of 510\n",
      "Preprocessed file 493 out of 510\n",
      "Preprocessed file 494 out of 510\n",
      "Preprocessed file 495 out of 510\n",
      "Preprocessed file 496 out of 510\n",
      "Preprocessed file 497 out of 510\n",
      "Preprocessed file 498 out of 510\n",
      "Preprocessed file 499 out of 510\n",
      "Preprocessed file 500 out of 510\n",
      "Preprocessed file 501 out of 510\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Preprocessed file 502 out of 510\n",
      "Preprocessed file 503 out of 510\n",
      "Preprocessed file 504 out of 510\n",
      "Preprocessed file 505 out of 510\n",
      "Preprocessed file 506 out of 510\n",
      "Preprocessed file 507 out of 510\n",
      "Preprocessed file 508 out of 510\n",
      "Preprocessed file 509 out of 510\n",
      "Preprocessed file 510 out of 510\n"
     ]
    }
   ],
   "source": [
    "# Get masks in mask directory\n",
    "mask_files = [os.path.join(mask_dir,f) for f in os.listdir(mask_dir) if os.path.isfile(os.path.join(mask_dir,f)) and 'nii.gz' in f]\n",
    "\n",
    "# Walk through ROI mask directory\n",
    "for mask_file in mask_files:\n",
    "    preproc_files = glob(preproc_dir + '*_preproc.nii.gz')\n",
    "    for pnum, preproc in enumerate(preproc_files):\n",
    "        print('Preprocessed file %d out of %d' % ((pnum + 1), len(preproc_files)))\n",
    "        items = preproc.split('_')\n",
    "        ses = items[-3]\n",
    "        run = items[-2]\n",
    "        subname = items[0].split('/')[-1]\n",
    "        event_file = glob(os.path.join(events_dir,subname,ses,'func','*' + run + '_events.tsv'))[0]\n",
    "        # Load events and image\n",
    "        events = pd.read_csv(event_file, sep = '\\t')\n",
    "        img = nib.load(preproc).get_fdata()\n",
    "        mask = nib.load(mask_file).get_fdata().astype(bool)\n",
    "        # Apply mask\n",
    "        roi = img[mask] # Shape: voxels x TRs\n",
    "        # Get relevant time intervals and labels from events file\n",
    "        for index, row in events.iterrows():\n",
    "            # Beginning TR of trial\n",
    "            start = int(round(row['onset']) / 2)\n",
    "            # Ending TR of trial, start + 10 sec, or 5 TRs\n",
    "            end = start + 5\n",
    "            x = roi[:,start:end].T\n",
    "            X.append(x) # Big X should be of shape (samples, timepoints, features)\n",
    "            Y.append(row['ImgName'])\n",
    "            sample2image2session = sample2image2session.append({'SampleIndex': len(X) - 1, \n",
    "                                                                'ImgName': row['ImgName'], \n",
    "                                                                'Subject': subname,\n",
    "                                                                'Session': int(ses[-2:]),\n",
    "                                                                'Run': int(run[-2:])}, ignore_index = True)\n",
    "        # Save last ten TRs as no stimulus, if enough data is left\n",
    "        if roi.shape[1] - end >= 5:\n",
    "            x = roi[:,end:end+5].T\n",
    "            X.append(x)\n",
    "            Y.append('none')\n",
    "            sample2image2session = sample2image2session.append({'SampleIndex': len(X) - 1, \n",
    "                                                                'ImgName': 'none', \n",
    "                                                                'Subject': subname,\n",
    "                                                                'Session': int(ses[-2:]),\n",
    "                                                                'Run': int(run[-2:])}, ignore_index = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Save the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "\n",
    "with open(data_dir + 'X_raw.p', 'wb') as f:\n",
    "    pickle.dump(X, f)\n",
    "    \n",
    "with open(data_dir + 'Y_raw.p', 'wb') as f:\n",
    "    pickle.dump(Y, f)\n",
    "\n",
    "with open(data_dir + 'sample2session.p', 'wb') as f:\n",
    "    pickle.dump(sample2image2session, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Process data from Shen et al. 2019, \"Deep image reconstruction from human brain activity\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"\n",
    "PARAMETERS - change these if you wish\n",
    "\n",
    "fmri_dir: where the fmriprep output data lives, should contain subject folders\n",
    "nuisance_dir: where the confounds_regressors.tsv files are located, should contain subject folders\n",
    "regressors_dir: an output directory to hold the nuisance regressors text files\n",
    "preproc_dir: an output directory for the fully processed subject data\n",
    "\n",
    "\"\"\"\n",
    "fd_threshold = 0.5 # Threshold of FD for censoring a frame\n",
    "# All the nuisance regressors we wish to remove. Do not include derivatives, these are calculated manually\n",
    "regressors = ['CSF', 'WhiteMatter','X','Y','Z','RotX','RotY','RotZ'] \n",
    "# Set directories\n",
    "fmri_dir, nuisance_dir, regressors_dir, preproc_dir, standard = ['../deepImageRecon/ds001506-download/',\n",
    "                                                                 '../deepImageRecon/ds001506-download/',\n",
    "                                                                 '../deepImageRecon/regressors/',\n",
    "                                                                 '../deepImageRecon/preprocessed/',\n",
    "                                                                 '/usr/share/fsl/5.0/data/standard/MNI152_T1_2mm_brain.nii.gz']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing sub-01\n",
      "\tSession 1 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 2 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 3 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 4 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 5 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 6 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 7 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 8 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 9 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 10 out of 18\n",
      "\t\tRun 1 out of 4\n",
      "\t\tRun 2 out of 4\n",
      "\t\tRun 3 out of 4\n",
      "\t\tRun 4 out of 4\n",
      "\tSession 11 out of 18\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 12 out of 18\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 13 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 14 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 15 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 16 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 17 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 18 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "Processing sub-02\n",
      "\tSession 1 out of 18\n",
      "\t\tRun 1 out of 10\n",
      "\t\tRun 2 out of 10\n",
      "\t\tRun 3 out of 10\n",
      "\t\tRun 4 out of 10\n",
      "\t\tRun 5 out of 10\n",
      "\t\tRun 6 out of 10\n",
      "\t\tRun 7 out of 10\n",
      "\t\tRun 8 out of 10\n",
      "\t\tRun 9 out of 10\n",
      "\t\tRun 10 out of 10\n",
      "\tSession 2 out of 18\n",
      "\t\tRun 1 out of 6\n",
      "\t\tRun 2 out of 6\n",
      "\t\tRun 3 out of 6\n",
      "\t\tRun 4 out of 6\n",
      "\t\tRun 5 out of 6\n",
      "\t\tRun 6 out of 6\n",
      "\tSession 3 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 4 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 5 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 6 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 7 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 8 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 9 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 10 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 11 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 12 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 13 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 14 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 15 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 16 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 17 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 18 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "Processing sub-03\n",
      "\tSession 1 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 2 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 3 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 4 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 5 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 6 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 7 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 8 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 9 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 10 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 11 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 12 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 13 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 14 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 15 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 16 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 17 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n",
      "\tSession 18 out of 18\n",
      "\t\tRun 1 out of 8\n",
      "\t\tRun 2 out of 8\n",
      "\t\tRun 3 out of 8\n",
      "\t\tRun 4 out of 8\n",
      "\t\tRun 5 out of 8\n",
      "\t\tRun 6 out of 8\n",
      "\t\tRun 7 out of 8\n",
      "\t\tRun 8 out of 8\n"
     ]
    }
   ],
   "source": [
    "# Get all subject directories\n",
    "subj_dirs = get_subj_dirs(fmri_dir)\n",
    "\n",
    "# Loop through each subjects and get regressors, perform scrubbing\n",
    "for subj in sorted(subj_dirs):\n",
    "    print('Processing %s' % subj)\n",
    "    # Absolute path of current subject\n",
    "    subj_dir_abs = os.path.join(fmri_dir, subj)\n",
    "    sess_dirs = sorted([f for f in os.listdir(subj_dir_abs) if os.path.isdir(os.path.join(subj_dir_abs, f)) and 'ses-' in f and 'NaturalImage' in f])\n",
    "    if not sess_dirs: # If there are not multiple sessions, then set to list of empty string to iterate only once in for loop\n",
    "        sess_dirs = ['']\n",
    "    for sessnum,sess in enumerate(sess_dirs):\n",
    "        print('\\tSession %d out of %d' % ((sessnum + 1), len(sess_dirs)))\n",
    "        # Absolute path of current session\n",
    "        sess_dir_abs = os.path.join(subj_dir_abs, sess)\n",
    "        conf_sess_dir_abs = os.path.join(nuisance_dir, subj, sess)\n",
    "        bold_files = sorted(glob(sess_dir_abs + '/func/*task-perception*bold.nii.gz'))\n",
    "        # For multiple runs\n",
    "        for runnum, bold in enumerate(bold_files):\n",
    "            print('\\t\\tRun %d out of %d' % ((runnum + 1), len(bold_files)))\n",
    "            # Perform registration\n",
    "            prefix = os.path.join(preproc_dir, subj + '_' + sess + '_run-' + str(runnum+1).zfill(2) + '_')\n",
    "            outfile = prefix + 'preproc.nii.gz'\n",
    "            outmat = os.path.join(sess_dir_abs,'func',subj + '_' + sess + '_run-' + str(runnum+1).zfill(2) + '_native2template.mat')\n",
    "            cmd = 'flirt -in ' + bold + ' -ref ' + standard + ' -omat ' + outmat\n",
    "            subprocess.call(cmd, shell = True)\n",
    "            cmd = 'flirt -in ' + bold + ' -ref ' + standard + ' -out ' + outfile + ' -init ' + outmat + ' -applyxfm -v'\n",
    "            subprocess.call(cmd, shell = True)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}