Core Analysis

#Overview The sbRIO sends data via a network stream that is written as .dat files on a 1 ms time scale by Labview to the appropriate directory. The raw data is organized first by Box number and Mouse ID, and then by contingency/date. (Contingency refers to the training protocol in place - i.e., the required parameters the mouse must satisfy for its training to obtain a reward).

#Data Storage

File Organization (Raw Data)

Each experiment is organized as follows:

expt_dir\data\Box_<i>_<Mouse ID>\

We use the following format for contingency codes

<thresh>_<holdtime>_<centerholdthresh>_<minangle>_<maxangle>

and the format mmddyy for dates. Here thresh refers to the outer threshold the mouse must move past and centerholdthresh is the inner threshold the mouse must keep the joystick inside for duration holdtime. minangle and maxangle define the sector that the mouse must move the joystick towards. Prior to post processing, data is organized as follows:

expt_dir\Box_<i>\<startdate>_<contingencycode>\*.dat

All raw data files are placed in a directory that combines date and contingency information by Labview. They are in the format Box_.dat. A single .dat file contains exactly 1 second of data, with data collected on a millisecond temporal scale.

One of the first steps in post processing is to combine multiple sequential days of the same contingency. After this is done, the organization looks like:

expt_dir\data\Box_<i>\<startdate>_<contingencycode>\<date>\*.dat

The date <startdate> is the date the contingency began, and the second date folder is subgrouping of all .dat files by day. For example, suppose we have the following data from a few days:

expt_dir\data\Box_2\072115_1_50_100_-180_180\*.dat
expt_dir\data\Box_2\072215_1_50_100_-180_180\*.dat
expt_dir\data\Box_2\072315_1_50_100_-180_180\*.dat
expt_dir\data\Box_2\072415_1_80_100_-180_180\*.dat
expt_dir\data\Box_2\072515_1_50_100_-180_180\*.dat

After the first step of combining contingency information, the organization will look like the following:

expt_dir\data\Box_2\072115_1_50_100_-180_180\072115\*.dat
expt_dir\data\Box_2\072115_1_50_100_-180_180\072215\*.dat
expt_dir\data\Box_2\072115_1_50_100_-180_180\072315\*.dat
expt_dir\data\Box_2\072415_1_80_100_-180_180\072415\*.dat
expt_dir\data\Box_2\072515_1_50_100_-180_180\072515\*.dat

Note that after grouping by contingency, each day is two levels down from the Box directory, not one. Secondly, contingency grouping still preserves chronological order, and doesn't group days that are the same contingency if they are separated by another contingency.

File Organization (Processed Data)

If these next couple sections describing the file organization system are confusing, see the section below on the Post Processing Pipeline first.

The last folder is the date of data collection. Before post processing, the folder will only contain .dat files. After processing, that folder will contain processed data as well the combined MATLAB .mat data files in the directory:

expt_dir\Box_i\<startdate>_<contingencycode>\<date>\comb\*.mat

The files

expt_dir\Box_i\<startdate>_<contingencycode>\<date>\jstruct.mat
expt_dir\Box_i\<startdate>_<contingencycode>\<date>\stats.mat

are also in the directory now.

Automated Contingency Updates

This feature is currently not supported by our LabView code, though the MATLAB component supports the filewriting procedure described below. Contingency information is accessible through the folder name, but to allow communication between MATLAB and Labview, we keep separate text files as well. Contingency information for each box is stored as follows:

expt_dir\Box_<i>\contingency.txt
expt_dir\Box_<i>\ArchivedContingencies\contingency_mmddyy_HHMM.txt

ArchivedContingencies stores the old text files, adding a timestamp of when they were changed - so a contingency ran from the timestamp of the previous archived text file, to its timestamp. It isn't necessary usually to access this folder except for debugging. The contingency code in the contingency folder name allows indexing days by contingency more efficiently than determining the current contingency based on time stamps.

Each contingency file has the format:

Box Num <number>
Out Threshold <number>
Hold Duration <number>
Hold Threshold <number>
Min Angle <number>
Max Angle <number>

Each item is separated by a space. MATLAB writes Box number as a double. Hold Threshold is actually Center Hold Threshold and Out Threshold is the same as Threshold (the one that starts at 1 at the beginning of the training protocol). (MATLAB parsing is easier with the exact same number of words for each description).

#Post Processing Pipeline

Raw .dat files in contingency-date folders 
--combine_contingency-->
data in date folders in contingency folders 
--ppscript-->
combined .mat files 
--xy_makestruct-->
jstruct 
--xy_getstats-->
stats

• combine_contingency combine_contingency (x) moves all raw .dat files in the directory x to a new directory, assuming x is in the format _/*.dat.

• ppscript: ppscript(x) processes all the individual .dat files and combines them into .mat files, placing them into a subdirectory comb in dir x. These combined .mat files represent a single "bout" of data collection, at least 3 seconds long, but sometimes as long as 1-2 minutes.

• xy_makestruct: After this step is executed, xy_makestruct(x/comb) takes all the combined matlab files and creates a single struct summarizing all the data (called jstruct). For specifics, see PostProcessing/CoreAnalysis/xy_makestruct about the fields the jstruct contains. The jstruct is organized/indexed by data bout.

• xy_getstats stats = xy_getstats(jstruct) processes the jstruct and returns a structure with statistics information. Most scripts use the stats structure. Postprocessing scripts use both the jstruct and stats structures as arguments - but never the raw .dat or even .mat files (at least not currently). See the script PostProcessing/CoreAnalysis/xy_getstats documentation for info about the stats structures fields.

• doAllpp: doAllpp(x) takes a directory and executes the entire pipeline, though an option exists to select the starting step. It combines contingencies, combines the raw .dat files, creates the jstruct, and the stats structures. After this step is executed, any number of Post Processing routines can be called.

• multi_doAll: multi_doAll(dirlist) does exactly what doAllpp does, but for all directories in dirlist, and generates an analysis report when it does so.