Full movie: YouTube
This repository contains code for LEAP (LEAP Estimates Animal Pose), a framework for animal body part position estimation via deep learning.
Preprint: Pereira et al., bioRxiv (2018)
We are still working on documentation and preparing parts of the code. See the Features section below for an overview and status of each component.
We recommend starting with the Tutorial: Training Leap From Scratch.
- Tracking and alignment code
- Cluster sampling GUI
- Skeleton creation GUI (
- GUI for labeling new dataset (
- Network training through the labeling GUI
- MATLAB (
predict_box.m) and Python (
leap.predict_box) interfaces for predicting on new data
- GUI for predicting on new data
- Training data + labels for main fly dataset used in analyses
- Trained network for predicting on main fly dataset
- Analysis/figure generation code
- Examples of usage
All neural network and GPU functionality is implemented in Python. The library was designed to be easy to use by providing commandline interfaces, but it can also be used programatically if the MATLAB GUIs are not required.
For the Python environment, we recommend Anaconda 5.1.0 with Python 3.6.4. Note that Python 2.x is not supported.
For GPU support, you'll want to first install the CUDA drivers with CuDNN and then install these packages:
pip install -Iv tensorflow-gpu==1.6.0 pip install -Iv keras==2.1.4
See the TensorFlow installation guide for more info.
If you don't have a GPU that supports CUDA, install the regular TensorFlow distribution:
pip install tensorflow
Please note that CPU execution will be MUCH slower (10-20x) than on a GPU. Consider investing in a GPU for your machine if you're thinking of using LEAP routinely.
To get started with using LEAP, open up MATLAB and download the repository:
>> !git clone https://github.com/talmo/leap.git
Then, install the package and add to the MATLAB path:
>> cd leap >> install_leap
To avoid having to run this function every time you start MATLAB, save the MATLAB Search Path after running it (or just add the
leap subfolder to your permanent path).
Manual: MATLAB dependencies
GUIs and analyses are implemented in MATLAB, but is not required for using the neural network functionality implemented in Python.
We use MATLAB R2018a with the following toolboxes: Parallel Computing Toolbox, Statistics and Machine Learning Toolbox, Computer Vision Toolbox, Image Processing Toolbox, Signal Processing Toolbox.
All MATLAB external toolboxes are included in the
leap/toolbox subfolder. Just add the
leap subdirectory to the MATLAB Search Path to access all functionality:
Manual: Python dependencies
The versions below were used during development of LEAP but other versions will also likely work.
Libraries required are easily installable via the pip package manager:
pip install -Iv numpy==1.14.1 pip install -Iv h5py==2.7.1 pip install -Iv clize==4.0.3
You will also need OpenCV 3 with Python bindings. We recommend using skvark's excellent precompiled packages:
pip install -Iv opencv-python==220.127.116.11
You can install this library as a python package by downloading this git repository:
git clone https://github.com/talmo/leap.git
pip install -e leap # installs the leap directory using pip
If you are using anaconda to manage different python environments, it is highly recommended that you use this matlab script to manage your python envs within matlab: condalab (see their readme for how to use it)
Refer to the Tutorial: Training Leap From Scratch.
- Cluster sampling: Call
cluster_samplefrom MATLAB commandline to launch GUI.
- Create skeleton: Call
create_skeletonfrom MATLAB commandline to launch GUI.
- Label data and train: Call
label_jointsfrom MATLAB commandline to launch GUI.
- Batch estimation: Coming soon.
leap/predict_box.py for more info.
Contact and more information
Reach out to us via email: Talmo Pereira (