Skip to content

Home

Jump to bottom
kristinquick edited this page Aug 25, 2019 · 34 revisions

Project Outline

Diagram of the planned additions to the data analysis pipeline for NWB:N 2.0 integration. Blue: C++, orange: MATLAB, purple: python.

Neuropixels are recorded using spikeGLX. This results in proprietary binary files that are fed into Kilosort2 for off-line spike sorting. Kilosort outputs the data as .mat files, which are then manually inspected using phy for curation. You then use custom scripts in MATLAB to synchronize the processed neural data with the behavioral events. Finally, this data is fed to Spikes for common analyses.

Part One

NWB:N integration will be accomplished by building a custom python program to convert from the output of the synchronization step to NWB:N 2.0. This will allow the lab to archive and share processed data in NWB:N 2.0 will facilitate development of tools that work directly with NWB:N 2.0 files, and will allow the lab to access the growing set of tools emerging in the community that work directly with NWB:N 2.0 files. We will demonstrate the usage of these tools for neuropixel data.

  • create python project (giocomo-lab-to-nwb)
  • load .mat file into python
  • create NWB file
    • create GUI for user to input general experiment metadata before running file conversion
    • update GUI code
      • datetimes need to be in ISO 8601. pynwb can convert it for you if you input a datetime object. A string field won't work. Use something like https://pypi.org/project/tkcalendar/.
      • use a numpy docstring for def convert
      • unpack results of tuple into variables in conversion.py by doing convert(*button_run_output_tuple)
    • insert user input data from GUI into nwb file
    • create README with GUI photo
    • conversion options:
      • GUI
      • single .mat file
      • batch convert .mat files
    • trial information
    • virtual position data
    • physical position data
    • electrode information
      • initial electrode information from .mat file
      • update x and y coordinates with surgical implant location They do not have ABA coordinates, only stereotaxic coordiantes.
      • add another column for electrode location on the probe ("relativex" and "relativey")
    • lick information
    • Units table for manually-adjusted units (clusters)
    • Units table for automatically-sorted units (spike_templates)
    • insert whitening matrix into NWB file
    • incorporate "tempScalingAmps" vector into Units table for automatically-sorted units
    • make corrections
      • for the position timeseries objects and for any timeseries with a constant sampling rate, use rate and starting_time instead of timestamps
      • trial contrast being stored as vector, change to constant
      • the ProcessingModule that holds TemplateUnits should be named 'ecephys'
    • validate NWB file
      • create .nwb file and upload to drive/giocomo/data folder
      • load .nwb file into MATLAB
  • create final version of metadata documentation
  • add metadata documentation to wiki
  • documentation / docstrings
  • update Jupyter widget for visualizing spikes vs time
    • make giocomo_vis_spec the default
    • add tabs in behavior for plotting the speed through maze
    • make unit's rasters color by depth
  • Add default values for def convert(), use dictionary instead of tuple and unpack with **
    • update interface_gui.py
    • update interface_config.py

Part Two

Build a custom MATLAB interface that allows you to load NWB:N files directly into Spikes. Spikes will load NWB:N files into the same in-memory objects as the lab currently uses. This will allow the lab to analyze NWB:N files with the same workflow that they currently use. We will attempt to work with the spikes development team to incorporate this tool directly into their repository and demonstrate usage of these tools on NWB:N data.

  • obtain sample code of how Giocomo lab uses their standard .mat file with Spikes. Spikes documentation leads me to believe it takes .npy files, so I would like to see what the Giocomo lab does to work in their synchronized .mat file into Spikes.
  • Load .nwb file into spikes
    • Read .nwb into MATLAB and convert back into the variables in the workspace that are read by spikes:
      • lickt, lickx
      • post, posx
      • sp
      • trial
      • trial_gain
      • trial contrast
    • Make a wrapper function that will check filetype of lfpFilename in memmapfile (.dat or .nwb) and if:
      • .dat - continue loading data into mmf variable as orginally done
      • .nwb - use hdf5's memmap capabilities to lazy-read data from .nwb into mmf variable

Mid-way Check-in

We will report our progress in an in-person lab visit where we present our progress to the point-person as well as other interested stake-holders in the lab. We will solicit feedback from the lab, and ensure that our progress is in line with the expectations and goals of the lab. We will provide the slides from this presentation to the Simons Foundation.

  • plan meeting day
  • book flight
  • book hotel
  • create presentation
    • create a slide with a diagram showing how we plan to use the spikeextractor tool to convert data into NWB and to interface NWB with a number of different spike sorting tools
    • show ephys-vis

Part Three

Build a python programmatic interface that writes the binary outputs of spikeGLX into the NWB:N file. Due to the size of the acquired data, we will not be able to hold the entire dataset in memory, so this will require us to build a mechanism within the MatNWB API to write data in chunks. This mechanism does not currently exist in MATLAB, and building this mechanism will take extra time, so we are not able to guarantee the completion of this component within the scope of this pilot project. This aim requires access to an example spikeGLX acquisition dataset.

Wrapping up

Using this feedback, we will finish building conversion program in Matlab, creating and publicly releasing any necessary NWB extensions. We will provide the lab with a tutorial showing them how to use this program and extensions. The program will be used to convert one example dataset to NWB:N. We will also discuss any final adjustments and any potential future projects with the lab. We will also provide the slides from this presentation to the Simons Foundation.

  • plan meeting day
  • book flight
  • book hotel
  • publically release giocomo-lab-to-nwb
  • create tutorial to show how to use program and jupyter notebook extension
  • create presentation slides
Clone this wiki locally