Skip to content
SpikeForest -- spike sorting analysis for website
Branch: master
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Permalink
Type Name Latest commit message Commit time
Failed to load latest commit information.
.vscode/snippets
bin
devel
docs
gui
misc/simplot
mountaintools
spikeforest
working
.codepod.yml
.gitignore
.travis.yml
LICENSE
README.md
codepod_run.sh
pytest.ini
run_pytest.sh

README.md

SpikeForest

Build Status

Note: This project is in alpha stage of development

Overview

This software supports the SpikeForest web site (in progress) for public validation and comparison of spike sorting algorithms applied to an expanding collection of hosted electrophysiology recordings with ground-truth spiking information. But you can also use this project for your own spike sorting, using MountainSort, IronClust, and other algorithms.

The framework that supports the requirements of the website, including wrapping of spike sorters in singularity containers and python classes, job batching, comparison with ground truth, and processing using remote compute resources, is all open source and may be of benefit to neuroscience labs.

We make use of the SpikeInterface project, also in alpha development stage, that provides tools for extracting, converting between, and curating raw or spike sorted extracellular data from any file format.

This repository is split into two pieces: MountainTools and SpikeForest. The former is not spike-sorting specific and can be used for other applications or downstream analyses. It provides modules for batch processing, automatic caching of results, sharing data between labs (from a python interface), and processing on remote compute resources. The latter contains wrappers to the spike sorters, analysis routines, GUI components, and the actual scripts used to prepare the data hosted on the website.

The code for the front-end website is also open source, but is hosted in a separate repository.

Installation

It is recommended that you start from a fresh conda environment with python (>= 3.6) installed.

pip install git+https://github.com/flatironinstitute/spikeforest#subdirectory=spikeforest
pip install git+https://github.com/flatironinstitute/spikeforest#subdirectory=mountaintools

If you want to use the containerized versions of the spike sorters, you should install singularity. link to instructions Otherwise you will need to install the sorters individually. You can install MountainSort4 via

pip install ml_ms4alg

In the future we will provide conda packages for these.

More advanced installation instructions for Windows 10 users and developers can be found below.

Installing IronClust

  • Clone IronClust from Github by running git clone https://github.com/jamesjun/ironclust
  • Add export IRONCLUST_PATH=install_location in .bashrc

Installing KiloSort

  • Clone KiloSort from Github by running git clone https://github.com/cortex-lab/KiloSort
  • Add export KILOSORT_PATH=install_location in .bashrc

Installing Kilosort2

  • Clone Kilosort2 from Github by running git clone https://github.com/MouseLand/Kilosort2
  • Add export KILOSORT2_PATH=install_location in .bashrc

Basic usage

Basic flow chart - SpikeForest

There are various ways to use MountainTools and SpikeForest as illustrated in the above diagram. Here we describe a few of the use cases.

Sorting a single recording

We start with the simplest case of a single recording.

  • Step 1: prepare your recording in mda format

[pull text for elsewhere here]

  • Step 2: run spike sorting
from spikesorters import MountainSort4

MountainSort4.execute(
    recording_dir=<recording directory>,
    firings_out=<output directory>/firings.mda,
    detect_sign=-1,
    adjacency_radius=50,
    _container=None
)

By specifying _container=None the system will attempt to run the spike sorting using software installed on your system, so you would need to have the ml_ms4alg python package installed. If you instead specified _container='default', the appropriate singularity container would be automatically downloaded. In that case you would need to have singularity installed.

Example notebook: example_single_recording.ipynb

Sorting with any spike sorter

You can use spikeforest to run other spike sorters as well. For example, to run spyking circus, first follow the preliminary steps above, and then:

from spikesorters import SpykingCircus

SpykingCircus.execute(
    recording_dir=<recording directory>,
    firings_out=<output directory>/firings.mda,
    detect_sign=-1,
    adjacency_radius=50,
    _container='default' # To fetch the default container for spyking circus
)

As described above, by using _container='default' you do not need to install spyking circus, although if you do have spyking circus on your machine, you could simply use _container=None.

Other spike sorters in progress include YASS, KiloSort, and IronClust.

Sorting batches of multiple recordings

[TODO: write this]

Example notebook: example_multi_recording.ipynb

Sorting using a remote compute resource

[TODO: write this]

Visualization of recordings and sorting results

[TODO: write this]

Data sharing

[TODO: write this]

For developers and testers: Opening in codepod (containerized VS Code IDE)

You can use and/or develop SpikeForest2 with codepod. Tested in Linux, should also work on a Mac but probably requires some tweaks.

Prerequisites: docker and codepod

First clone this repo:

git clone https://github.com/flatironinstitute/spikeforest

Next, set the KBUCKET_CACHE_DIR environment variable. This is where the cached files from kbucket will go. For example, you could use export KBUCKET_CACHE_DIR=/tmp/sha1-cache

Install codepod

pip install --upgrade codepod

Then run the ./codepod_run.sh convenience script in the repo

cd spikeforest
./codepod_run.sh

This will download a docker image (may take some time depending on the speed of your internet connection) and put you in a container with a fully-functional development environment.

Once inside the container you can run the following to open vscode

code .

Unit tests

Once in codepod, you may run the unit tests via

pytest

To run the slower, more thorough, tests:

pytest -m slow -s
# The -s flag is for verbose output, which may not be what you want
# This will download singularity containers, which may take some time
# depending on your internet connection

Installation for developers (without using codepod)

Once installing dependencies below, run . devel/setup_python.sh from the main spikeforest directory to setup the conda environment.

  • Default conda environment is named to spikeforest unless you specify an argument (e.g. . devel/setup_python.sh my_env)
  • The script installs all necessary dependencies for developing and using SpikeForest.

Dependencies

  1. CONDA: setup_python.sh uses conda installed in ~/conda/etc/profile.d/conda.sh. Change this line to point to the correct conda location (see your ~/.bashrc).
  2. ml_ms4alg requires g++ installation which may not be part of your OS. If this is the case, run sudo apt install build-essential.
  3. nodejs to use ipywidgets in jupyter notebooks. To install nodejs, run sudo apt install nodejs npm -y.
  4. Install docker and singularity

Installation in Windows 10 using Ubuntu WSL

  1. Install Windows Subsystem for Linux (WSL). Run Turn Windows features on or off and check Windows Subsystem for Linux
  2. Install Ubuntu 18.04 though the Microsoft store app
  3. Start Ubuntu and install Miniconda for Linux
  4. Run sudo apt install build-essential nodejs npm -y
  5. Modify devel/setup_python.sh to point to the correct conda.sh (e.g. ~/miniconda3/etc/profile.d/conda.sh)
  6. Run devel/setup_python.sh
  7. Test if installed correctly by running one of the example notebook. Run conda activate spikeforest and run jupyter lab --no-browser and copy the link. Paste in Chrome browser in Windows

Authentication Configuration

Some advanced operations (for example processing on a remote compute resource) requires authentication. You can obtain the ID and password from Jeremy Magland and then create a text file in .mountaintools/.env where you can set the MOUNTAIN_USER and MOUNTAIN_PASSWORD fields:

MOUNTAIN_USER=SPRUCE
MOUNTAIN_PASSWORD=TREE

Directory structure

(Please notify if the following gets out-of-sync with the project directory structure)

devel: Utilities specific to development of SpikeForest, including instructions on preparing the docker image for codepod, a script to run when codepod is started, and a script for auto-formatting the python code for pep8 compliance.

mountaintools: Contains the MountainTools such as cairio, kbucket, mlprocessors, and vdomr. These tools are not specific to spike sorting.

spikeforest/spikesorters: Wrappers of the spike sorting algorithms, including Dockerfile recipes for the containers.

working: The SpikeForest analysis scripts. Contains scripts for preparing recordings, running spike sorting, comparing with ground truth, and assembling results for the websites.

spikeforest/spikeforest_analysis: A python module using by spike sorting scripts and analysis scripts. Contains the core processing routines.

spikeforest/spikeforestwidgets: Some vdomr widgets used by the GUIs.

.codepod.yml: Configuration file for codepod

.gitignore: Files that git should ignore

codepod_run.sh: Run this to open the project using codepod (see above).

pytest.ini: Configuration file for pytest

README.md: This readme file

LICENSE: The license file for this project. See also license files for individual components within subdirectories.

You can’t perform that action at this time.