Note: This project is in alpha stage of development
This software supports the SpikeForest web site (in progress) for public validation and comparison of spike sorting algorithms applied to an expanding collection of hosted electrophysiology recordings with ground-truth spiking information. But you can also use this project for your own spike sorting, using MountainSort, IronClust, and other algorithms.
The framework that supports the requirements of the website, including wrapping of spike sorters in singularity containers and python classes, job batching, comparison with ground truth, and processing using remote compute resources, is all open source and may be of benefit to neuroscience labs.
We make use of the SpikeInterface project, also in alpha development stage, that provides tools for extracting, converting between, and curating raw or spike sorted extracellular data from any file format.
This repository is split into two pieces: MountainTools and SpikeForest. The former is not spike-sorting specific and can be used for other applications or downstream analyses. It provides modules for batch processing, automatic caching of results, sharing data between labs (from a python interface), and processing on remote compute resources. The latter contains wrappers to the spike sorters, analysis routines, GUI components, and the actual scripts used to prepare the data hosted on the website.
The code for the front-end website is also open source, but is hosted in a separate repository.
It is recommended that you start from a fresh conda environment with python (>= 3.6) installed.
pip install git+https://github.com/flatironinstitute/spikeforest#subdirectory=spikeforest pip install git+https://github.com/flatironinstitute/spikeforest#subdirectory=mountaintools
If you want to use the containerized versions of the spike sorters, you should install singularity. link to instructions Otherwise you will need to install the sorters individually. You can install MountainSort4 via
pip install ml_ms4alg
In the future we will provide conda packages for these.
More advanced installation instructions for Windows 10 users and developers can be found below.
IronClustfrom Github by running
git clone https://github.com/jamesjun/ironclust
KiloSortfrom Github by running
git clone https://github.com/cortex-lab/KiloSort
Kilosort2from Github by running
git clone https://github.com/MouseLand/Kilosort2
There are various ways to use MountainTools and SpikeForest as illustrated in the above diagram. Here we describe a few of the use cases.
Sorting a single recording
We start with the simplest case of a single recording.
- Step 1: prepare your recording in mda format
[pull text for elsewhere here]
- Step 2: run spike sorting
from spikesorters import MountainSort4 MountainSort4.execute( recording_dir=<recording directory>, firings_out=<output directory>/firings.mda, detect_sign=-1, adjacency_radius=50, _container=None )
_container=None the system will attempt to run the spike sorting using software installed on your system, so you would need to have the
ml_ms4alg python package installed. If you instead specified
_container='default', the appropriate singularity container would be automatically downloaded. In that case you would need to have singularity installed.
Sorting with any spike sorter
You can use spikeforest to run other spike sorters as well. For example, to run spyking circus, first follow the preliminary steps above, and then:
from spikesorters import SpykingCircus SpykingCircus.execute( recording_dir=<recording directory>, firings_out=<output directory>/firings.mda, detect_sign=-1, adjacency_radius=50, _container='default' # To fetch the default container for spyking circus )
As described above, by using
_container='default' you do not need to install spyking circus, although if you do have spyking circus on your machine, you could simply use
Other spike sorters in progress include YASS, KiloSort, and IronClust.
Sorting batches of multiple recordings
[TODO: write this]
Sorting using a remote compute resource
[TODO: write this]
Visualization of recordings and sorting results
[TODO: write this]
[TODO: write this]
For developers and testers: Opening in codepod (containerized VS Code IDE)
You can use and/or develop SpikeForest2 with codepod. Tested in Linux, should also work on a Mac but probably requires some tweaks.
First clone this repo:
git clone https://github.com/flatironinstitute/spikeforest
Next, set the KBUCKET_CACHE_DIR environment variable. This is where the cached files from kbucket will go. For example, you could use
pip install --upgrade codepod
Then run the ./codepod_run.sh convenience script in the repo
cd spikeforest ./codepod_run.sh
This will download a docker image (may take some time depending on the speed of your internet connection) and put you in a container with a fully-functional development environment.
Once inside the container you can run the following to open vscode
Once in codepod, you may run the unit tests via
To run the slower, more thorough, tests:
pytest -m slow -s # The -s flag is for verbose output, which may not be what you want # This will download singularity containers, which may take some time # depending on your internet connection
Installation for developers (without using codepod)
Once installing dependencies below, run
. devel/setup_python.sh from the main
spikeforest directory to setup the conda environment.
- Default conda environment is named to
spikeforestunless you specify an argument (e.g.
. devel/setup_python.sh my_env)
- The script installs all necessary dependencies for developing and using SpikeForest.
setup_python.shuses conda installed in
~/conda/etc/profile.d/conda.sh. Change this line to point to the correct conda location (see your
g++installation which may not be part of your OS. If this is the case, run
sudo apt install build-essential.
ipywidgetsin jupyter notebooks. To install
sudo apt install nodejs npm -y.
Installation in Windows 10 using Ubuntu WSL
- Install Windows Subsystem for Linux (WSL). Run
Turn Windows features on or offand check
Windows Subsystem for Linux
- Install Ubuntu 18.04 though the Microsoft store app
- Start Ubuntu and install Miniconda for Linux
sudo apt install build-essential nodejs npm -y
devel/setup_python.shto point to the correct conda.sh (e.g.
- Test if installed correctly by running one of the example notebook. Run
conda activate spikeforestand run
jupyter lab --no-browserand copy the link. Paste in Chrome browser in Windows
Some advanced operations (for example processing on a remote compute resource) requires authentication. You can obtain the ID and password from Jeremy Magland and then create a text file in
.mountaintools/.env where you can set the
(Please notify if the following gets out-of-sync with the project directory structure)
devel: Utilities specific to development of SpikeForest, including instructions on preparing the docker image for codepod, a script to run when codepod is started, and a script for auto-formatting the python code for pep8 compliance.
mountaintools: Contains the MountainTools such as cairio, kbucket, mlprocessors, and vdomr. These tools are not specific to spike sorting.
spikeforest/spikesorters: Wrappers of the spike sorting algorithms, including Dockerfile recipes for the containers.
working: The SpikeForest analysis scripts. Contains scripts for preparing recordings, running spike sorting, comparing with ground truth, and assembling results for the websites.
spikeforest/spikeforest_analysis: A python module using by spike sorting scripts and analysis scripts. Contains the core processing routines.
spikeforest/spikeforestwidgets: Some vdomr widgets used by the GUIs.
.codepod.yml: Configuration file for codepod
.gitignore: Files that git should ignore
codepod_run.sh: Run this to open the project using codepod (see above).
pytest.ini: Configuration file for pytest
README.md: This readme file
LICENSE: The license file for this project. See also license files for individual components within subdirectories.