Skip to content

Core Analysis

Jump to bottom Edit New page
Goldberg Lab Administrator edited this page Aug 21, 2015 · 7 revisions

#Overview The sbRIO sends data via a network stream that is written as .dat files on a 1 ms time scale by Labview to the appropriate directory. The raw data is organized first by Box number and Mouse ID, and then by contingency/date. (Contingency refers to the training protocol in place - i.e., the required parameters the mouse must satisfy for its training to obtain a reward).

#Data Storage

File Organization (Raw Data)

Each experiment is organized as follows:

expt_dir\Box_<i>_<Mouse ID>\

We use the following format for contingency codes

<thresh>_<holdtime>_<centerholdthresh>_<minangle>_<maxangle>

and the format mmddyy for dates. Here thresh refers to the outer threshold the mouse must move past and centerholdthresh is the inner threshold the mouse must keep the joystick inside for duration holdtime. minangle and maxangle define the sector that the mouse must move the joystick towards. Prior to post processing, data is organized as follows:

expt_dir\Box_<i>\<startdate>_<contingencycode>\*.dat

All raw data files are placed in a directory that combines date and contingency information by Labview. They are in the format Box_.dat. A single .dat file contains exactly 1 second of data, with data collected on a millisecond temporal scale.

One of the first steps in post processing is to combine multiple sequential days of the same contingency. After this is done, the organization looks like:

expt_dir\Box_<i>\<startdate>_<contingencycode>\<date>\*.dat

The date is the date the contingency began, and the second date folder is subgrouping of all .dat files by day. For example, suppose we have the following data from a few days:

expt_dir\Box_2\072115_1_50_100_-180_180\*.dat
expt_dir\Box_2\072215_1_50_100_-180_180\*.dat
expt_dir\Box_2\072315_1_50_100_-180_180\*.dat
expt_dir\Box_2\072415_1_80_100_-180_180\*.dat
expt_dir\Box_2\072515_1_50_100_-180_180\*.dat

After the first step of combining contingency information, the organization will look like the following:

expt_dir\Box_2\072115_1_50_100_-180_180\072115\*.dat
expt_dir\Box_2\072115_1_50_100_-180_180\072215\*.dat
expt_dir\Box_2\072115_1_50_100_-180_180\072315\*.dat
expt_dir\Box_2\072415_1_80_100_-180_180\072415\*.dat
expt_dir\Box_2\072515_1_50_100_-180_180\072515\*.dat

Note that after grouping by contingency, each day is two levels down from the Box directory, not one. Secondly, contingency grouping still preserves chronological order, and doesn't group days that are the same contingency if they are separated by another contingency.

File Organization (Processed Data)

If these next couple sections describing the file organization system are confusing, see the section below on the Post Processing Pipeline first.

The last folder is the date of data collection. Before post processing, the folder will only contain .dat files. After processing, that folder will contain processed data as well the combined MATLAB .mat data files in the directory:

expt_dir\Box_i\<startdate>_<contingencycode>\<date>\comb\*.mat

The files

expt_dir\Box_i\<startdate>_<contingencycode>\<date>\jstruct.mat
expt_dir\Box_i\<startdate>_<contingencycode>\<date>\stats.mat

are also in the directory now.

Automated Contingency Updates

This feature is currently not supported by our LabView code, though MATLAB computation already supports the filewriting procedure described below. Contingency information is accessible through the folder name, but to allow communication between MATLAB and Labview, we keep separate text files as well. Contingency information for each box is stored as follows:

expt_dir\Box_<i>\contingency.txt
expt_dir\Box_<i>\ArchivedContingencies\contingency_mmddyy_HHMM.txt

ArchivedContingencies stores the old text files, adding a timestamp of when they were changed - so a contingency ran from the timestamp of the previous archived text file, to its timestamp. It isn't necessary usually to access this folder except for debugging. The contingency code in the contingency folder name allows indexing days by contingency more efficiently than determining the current contingency based on time stamps.

Each contingency file has the format:

Box Num <number>
Out Threshold <number>
Hold Duration <number>
Hold Threshold <number>
Min Angle <number>
Max Angle <number>

Each item is separated by a space. MATLAB writes Box number as a double. Hold Threshold is actually Center Hold Threshold and Out Threshold is the same as Threshold (the one that starts at 1 at the beginning of the training protocol). MATLAB parsing is much easier with the exact same number of words for each description.

#Post Processing Pipeline

Raw .dat files in contingency-date folders 
--combine_contingency-->
data in date folders in contingency folders 
--ppscript-->
combined .mat files 
--xy_makestruct-->
jstruct 
--xy_getstats-->
stats

  • combine_contingency combine_contingency (x) moves all raw .dat files in the directory x to a new directory, assuming x is in the format _/*.dat.

  • ppscript: ppscript(x) processes all the individual .dat files and combines them into .mat files, placing them into a subdirectory comb in dir x. These combined .mat files represent a single "bout" of data collection, at least 3 seconds long, but sometimes as long as 1-2 minutes.

  • xy_makestruct: After this step is executed, xy_makestruct(x/comb) takes all the combined matlab files and creates a single struct summarizing all the data (called jstruct). For specifics, see PostProcessing/CoreAnalysis/xy_makestruct about the fields the jstruct contains. The jstruct is organized/indexed by data bout.

  • xy_getstats stats = xy_getstats(jstruct) processes the jstruct and returns a structure with statistics information. Most scripts use the stats structure. Postprocessing scripts use both the jstruct and stats structures as arguments - but never the raw .dat or even .mat files (at least not currently). See the script PostProcessing/CoreAnalysis/xy_getstats documentation for info about the stats structures fields.

  • doAllpp: doAllpp(x) takes a directory and executes the entire pipeline, though an option exists to select the starting step. It combines contingencies, combines the raw .dat files, creates the jstruct, and the stats structures. After this step is executed, any number of Post Processing routines can be called.

  • multi_doAll: multi_doAll(dirlist) does exactly what doAllpp does, but for all directories in dirlist, and generates an analysis report when it does so.

#Data Analysis Scripts Below is a summary of each analysis function's role. For detailed information and the full specification see the individual function.

###PPUtilities

  • rdir: returns a list of files or directories (supports arbitrary file matching - ex: rdir([pwd, '*']);
  • draw_heat_map: draws a heat map, supporting options for logarithmic mapping and color percentiles.
  • hist2d: computes a 2d histogram from 2 dimensional data.

###SensorDistributions Dependencies: requires stats struct to be saved in directory

  • gaps: plots the distribution of the duration a sensor was on
  • np_js_distribution: plots the nosepoke joystick onset distribution
  • np_post_distribution: plots the nosepoke post touch onset distribution
  • multipcolor_np_js: generates a pcolor plot of a single mouse's progress in the nosepoke joystick onset distribution for all days with data.
  • multipcolor_np_post: generates a pcolor plot of a single mouse's progress in the nosepoke post onset distribution for all days the mouse.

###Activity Heat Maps Dependencies: requires stats , velaccel, and velaccelraw structs to be saved in directory (generated by xy_getstats and compute_vel_accel_distr):

  • multiday_video: generates a video showing the progression of the activity heat map for multiple days
  • activity_summary_heat_maps: performs all of the analysis listed below on a single figure
  • activity_heat_map: heat map of joystick position during valid trajectories
  • velocity_heat_map: heat map of trajectory velocities using median
  • velocityvar_heat_map: heat map of trajectory velocity variation using difference between 75th and 25th percentiles for each cell.
  • accel_heat_map: heat map of trajectory acceleration at each point in (x,y) space using median measure
  • accelvar_heat_map: heat map of trajectory acceleration variation at each point in (x,y) space using difference between 75th and 25th percentiles
  • accelnorm_heat_map: heat map of the normal component of acceleration at each point in (x,y) space
  • accelang_heat_map: heat map of the angular component of acceleration at each point in (x, y) space
  • get_vel_accel_distr: loads information required for the above distributions/plots assuming that they have been saved to appropriate directories (data only). also combines data when appropriate rather than compute from scratch.
  • compute_vel_accel_distr: computes all information required for the above distributions/plots (data only) - can be very slow. does it separately for each day.

###Trajectory Analysis Dependencies: requires stats structs to be saved in directories:

  • trajectory_analysis: bins trajectories by hold time and then computes percentiles of trajectory distance at each time point
  • multi_trajectory_analysis: calls trajectory_analysis of magnitude of several days on one set of plots

###Time Distributions Dependencies: requires stats structs to be saved in directories (future edits may change this to improve efficiency, requiring a separate struct containing raw data instead)

  • hold_time_distr: plots distribution of hold times using js onset to js offset as measure of hold times
  • rewarded_time_distr: plots distribution of rewarded hold times
  • rewardrate_distr: plots distribution of reward rate by time interval
  • joystick_to_reward_distr: plots distribution of joystick touch onset to reward distribution

#Post Processing GUI The post processing GUI (pp_gui) is a tool to visualize multiple plots and perform data analysis. It allows selection of multiple different days and allows selection of any combination of plots. The process for making a new function and adding support for it to the GUI is described on the page Guidelines for New Analysis Scripts

Add a custom footer
Add a custom sidebar

Clone this wiki locally