External model inference

[sheridana]

Problem background

• There are various pretrained pose estimation models that could be integrated into SLEAP. It would be nice to provide a scaffolding to support these models.
• A good example to start with is MoveNet - a pose estimation model from Google.
• This would provide the added benefit of support for human pose estimation in SLEAP.
• This would also provide an example for supporting models from tensorflow hub

Feature proposal

• Add support for inference with pre-trained MoveNet model(s).

Implementation details

Example MoveNet inference

• Here is an example use of MoveNet for real-time pose matching
• This example has a wrapper function to load pretrained models which returns a keras model
• The keras model can then run inference by calling predict_on_batch, handled here for a single image

Corresponding SLEAP code

• The corresponding logic to load pretrained models in SLEAP is handled inside the predictor from_trained_models class method (example)
• This instantiates an InferenceModel (subclasses keras model) which also handles the prediction
• At a high level, the base Predictor class handles prediction via a generator which first takes care of data preprocessing and then runs inference

This PR

• Since there is a lot of specific logic in current predictor classes, it is probably best to create a separate predictor (e.g MoveNetPredictor) that subclasses the base Predictor class.
• This predictor will need to be exposed, similar to what is done for the other predictors
• The MoveNetPredictor should handle loading from tensorflow hub (rather than via tf.keras.models.load_model) as was done in the example
• We probably will want to override the pipeline creation, to support center padding as done in the example (or maybe this is already fine with the SizeMatcher?)
• It seems like MoveNet expects a batch size of 1, so this would need to be set appropriately in the predictor

[roomrys]

The PRs to integrate MoveNet have been merged; however, we still need to expose the MoveNet option to the GUI (and add support in the configuration files).

[roomrys]

This feature is now available in the (pre) release 1.3.0a0, to install, first uninstall and then:
conda (Windows/Linux/GPU):

conda create -y -n sleap -c sleap -c sleap/label/dev -c nvidia -c conda-forge sleap=1.3.0a0

pip (any OS except Apple Silicon):

pip install sleap==1.3.0a0

Warning: This is a pre-release! Expect bugs and strange behavior when testing.