This Python repository is adapted from the methods and code described in Mukherjee, Wachutka, & Katz (2017) [1]. A large percentage of the clustering parameters were made in reference to Reddish (2005) [2].
Heavily inspired by blechpy.
Note
This program is designed for sorting spikes from electrophysiological recordings into single, isolated units. The primary input is a .h5 file containing the continuous signal or thresholded waveforms. Alternatively, numpy arrays and pandas dataframes can be used with the help of data-loaders.
Compatibility and Licensing This software is compatible with Windows, macOS, and Linux, and is well-suited for containerization and high-performance computing clusters. It is distributed under the GNU General Public License v3.0 (GPLv3). For more information, consult the LICENSE file in this repository.
The primary criteria for considering a unit isolated are:
- 1 ms ISIs must be <= 0.5%
- The waveform must be cellular
- The unit must be sufficiently separated based on Mahalanobis distribution
- L-Ratio must be <= 0.1, as described in Schmitzer-Torbert et al. (2005) [2]_.
~/ ├── autosort │ ├── h5 │ ├── completed │ ├── results │ ├── to_run │ └── autosort_config.ini
To initialize a configuration file with default settings, use the default_config function as follows:
This pipeline requires Python 3.9+, and numpy <= 1.3.5 to comply with numba restrictions.
It is highly-recommended to install using mambaforge this will drastically speed up environment creation:
Linux and MacOS:
git clone https://github.com/FlynnOConnell/clustersort.git
cd path/to/clustersort
# This is for MambaForge, but you can use conda if you want
# Note if you use conda, but want to go the mamba route, you will really want to uninstall miniconda/anaconda first
wget -O Mambaforge.sh "https://github.com/conda-forge/miniforge/releases/latest/download/Mambaforge-$(uname)-$(uname -m).sh"
bash Mambaforge.sh -b -p "${HOME}/conda"
# !!!! FOR BASH USERS !!!!
# If you dont know what these are, then use this one
# If you use zsh, just change this to ~/.zshrc
echo ". ${HOME}/conda/etc/profile.d/conda.sh" >> ~/.bashrc
source "${HOME}/conda/etc/profile.d/conda.sh"If you're getting conda: command not found, you need to add conda to your path.
Look in your home directory, you should have a mambaforge or miniforge3 folder, depending on
your method of installation. Add that folder/bin to your path:
export PATH="/home/username/mambaforge/bin:$PATH"
mamba env create -f environment.yml # this will take a while
conda activate clustersort
pip install -r requirements.txt
pip install -e .Additionally, though not recommended, clustersort can be installed directly from pip:
Warning
pip installing has not been tested on systems other than linux.
Using mamba has been tested on each platform.
As has docker.
pip install clustersort. Mambaforge (recommended)
We recommend that you start with the Mambaforge distribution.
Mambaforge comes with the popular conda-forge channel preconfigured, but you can modify the configuration to use any channel you like.
Note that Anaconda channels are generally incompatible with conda-forge, so you should not mix them.
Note
For both mamba and conda, the base environment is meant to hold their dependencies.
It is strongly discouraged to install anything else in the base envionment.
Doing so may break mamba and conda installation.
. Existing conda install (not recommended)
Warning
This way of installing Mamba is not recommended. We strongly recommend to use the Mambaforge method (see above).
To get mamba, just install it into the base environment from the conda-forge channel:
# NOT RECOMMENDED: This method of installation is not recommended, prefer Mambaforge instead (see above)
# conda install -n base --override-channels -c conda-forge mamba 'python_abi=*=*cp*'Warning
Installing mamba into any other environment than base is not supported.
In addition to the Mambaforge standalone distribution (see above), there are also the condaforge/mambaforge docker images:
docker run -it --rm condaforge/mambaforge:latest mamba infoConfiguring the clustersort pipleine is done with an instance of SpkConfig:
import clustersort.SpkConfig
config = SpkConfig()You can set parameters with config.set():
config.set('run', 'resort-limit', '2')
print(config.run('resort-limit')
>>> '2'
print(type(config.run('resort-limit'))
>>> 'str'Warning
configuration parameters must always be sent as strings, and they will be returned as strings. It's up to the user to convert the type after retrieval.
- Sections
Configuration parameters for the runtime of the pipeline.
- resort-limit
- Description: The maximum number of times the sorting process can be rerun.
- Default: 3
- cores-used
- Description: The number of cores to be used during the run.
- Default: 8
- weekday-run
- Description: The number of runs allowed on a weekday.
- Default: 2
- weekend-run
- Description: The number of runs allowed on a weekend.
- Default: 8
- run-type
- Description: Defines the type of run (Auto/Manual).
- Default: Auto
- manual-run
- Description: The number of manual runs allowed.
- Default: 2
Here we define various paths necessary for the script, set by default to subdirectories in the parent directory of the specified path.
- run-path
- Description: Path to the directory where files to be processed are stored.
- Default: None specified
- results-path
- Description: Path to the directory where results will be stored.
- Default: None specified
- completed-path
- Description: Path where completed files will be moved.
- Default: None specified
Parameters defining the clustering process:
- max-clusters
- Description: Maximum number of clusters to use in the clustering algorithm.
- Default: 7
- max-iterations
- Description: Maximum number of iterations for the clustering algorithm.
- Default: 1000
- convergence-criterion
- Description: The criterion for convergence in the clustering algorithm.
- Default: .0001
- random-restarts
- Description: Number of random restarts in the clustering process to avoid local minima.
- Default: 10
- l-ratio-cutoff
- Description: The cutoff value for the L-Ratio metric, used to assess cluster quality.
- Default: .1
Parameters involved in signal preprocessing and spike detection:
- disconnect-voltage
- Description: Voltage level that indicates a disconnection in the signal, to detect noise or artifacts.
- Default: 1500
- max-breach-rate
- Description: The maximum rate at which breaches (potentially signal artifacts or spikes) can occur before it is considered noise.
- Default: .2
- max-breach-count
- Description: The maximum count of breaches allowed in a given window of time.
- Default: 10
- max-breach-avg
- Description: Perhaps the average breach level over a defined window.
- Default: 20
- intra-hpc_cluster-cutoff
- Description: A cutoff value for considering a signal as noise based on some intra-cluster metric.
- Default: 3
Filtering parameters to isolate the frequency range of interest:
- low-cutoff
- Description: The low cutoff frequency for a band-pass filter.
- Default: 600
- high-cutoff
- Description: The high cutoff frequency for the band-pass filter.
- Default: 3000
Spike detection and extraction parameters:
- pre-time
- Description: Time before a spike event to include in each spike waveform, in seconds.
- Default: .2
- post-time
- Description: Time after a spike event to include in each spike waveform, in seconds.
- Default: .6
- sampling-rate
- Description: The sampling rate of the recording, in Hz.
- Default: 20000
Standard deviation parameters for spike detection and artifact removal:
- spike-detection
- Description: A multiplier for the standard deviation of the noise to set a threshold for spike detection.
- Default: 2.0
- artifact-removal
- Description: A threshold for artifact removal, based on a multiple of the standard deviation.
- Default: 10.0
Parameters defining how principal component analysis (PCA) is conducted on the spike waveforms:
- variance-explained
- Description: The proportion of variance explained to determine the number of principal components to retain.
- Default: .95
- use-percent-variance
- Description: Whether to use percent variance to determine the number of components to retain.
- Default: 1
- principal-component-n
- Description: An alternative to variance-explained, specifying the number of principal components to retain directly.
- Default: 5
Post-processing parameters:
- reanalyze
- Description: Whether to reanalyze the data.
- Default: 0
- simple-gmm
- Description: Whether to use a simple Gaussian Mixture Model in the post-processing.
- Default: 1
- image-size
- Description: The size of images generated during post-processing.
- Default: 70
- temporary-dir
- Description: The directory to store temporary files during processing.
- Default: user's home directory followed by '/tmp_python'