Multiple Animal Experiments

Real-time, closed-loop Multiple Animal pose estimation

Disclaimer: Multiple Animal Experiments need to be custom designed for now.

There is an example experiment and trigger (ExampleSocialInteractionExperiment and SocialInteractionTrigger) available.

Go to the general design tutorial to learn how to design or adapt experiments to your own needs.

The necessary steps to upgrade your DLStream experience to real-time, closed-loop multiple animal experiments are written below!

Using two animals with different fur color and online SiMBA classification:

See the Advanced Behavior Classification section for further information how to integrate SiMBA classification into DLStream experiments. To get started with SiMBA classification, please refer to the main source: SiMBA and preprint

Full integration with SLEAP (pre-release Version)

Thanks to a great collaboration with the authors from SLEAP, which is available as a preprint on BioRxiv, we can now offer full integration of multiple animal (mA) experiments in DLStream.

How to install DLStream and SLEAP

2nd Disclaimer

Identity tracking is currently only possible with animals of different appearance (e.g. with different fur color) that can be tracked by regular models (e.g. from DLC-Live). We already included some mA pose estimation capability with an experimental integration of maDLC ,DeepPoseKit and our own deprecated mA tracker. However, in mA pose estimation, the identity between frames is not conserved without further computation. So far, this was possible using post-processing steps (see maDLC, id-Tracker.ai, SLEAP) that would take several frames into the past (and sometime the "future) into account. A luxury that "real-time" tracking does not always have. As soon as this is available, we will integrate it into DLStream. So stay tuned for the next release.

How to use multiple animal pose estimation in DLStream?

1. The easiest way to get a fully functional real-time mA experiment is by using the newest SLEAP models.

Please refer to the official guides on how to set up and train these models.

In settings.ini change the following parameters:

This is an example for the dataset provided by SLEAP. Please use your own model paths and body part names.

[Pose Estimation]
MODEL_ORIGIN = SLEAP
MODEL_PATH = D:\SLEAP\example_data\models\baseline_model.centroids, D:\SLEAP\example_data\models\baseline_model.topdown
...
ALL_BODYPARTS = head, thorax, abdomen, wingL, wingR, forelegL4, forelegR4, midlegL4, midlegR4, hindlegL4, hindlegR4, eyeL, eyeR

In utils/advanced_settings.ini change the following parameters:

[Streaming]
...
PASS_SEPARATE = False
ANIMALS_NUMBER = 2 #this should represent the expected maximum number of instances/animals in the experiment

[Pose Estimation]
FLATTEN_MA = FALSE
SPLIT_MA = FALSE
HANDLE_MISSING = pass #If you want to use the raw SLEAP output use "pass", if you want to use the default DLStream method use "skip"

You can read more on the settings on this page: Settings.

The HANDLE_MISSING parameter is an important decision to make. If you decide to use raw pose estimation to your experiment, be sure to handle any missing value (NaN) cases. Sometimes an incomplete skeleton might not make any difference, but a missing body part that is utilized for a trigger will result in errors or unexpected behavior!

In our new custom/ExampleSocialInteractionExperiment and custom/SocialInteractionTrigger missing values (NaN) are already accounted for.

BTW: FLATTEN_MA flattens any "real" multiple animal pose estimation into a flat representation. This is very usefull if you previously designed experiments with these kind of models in mind, but do not want to redesign the entire experiment.

2. Another way to enable multiple animal experiments is the use of single instance models (from SLEAP or DLC) that utilize the major differences of animals with different appearance (e.g. fur color) and create a "flat" skeleton, where animals are not differentiated (e.g. nose1, neck1, ..., nose2, neck2 ,...).

For this situation we have two options in utils/advanced_settings.ini

[Pose Estimation]
FLATTEN_MA = FALSE
SPLIT_MA = TRUE

SPLIT_MA splits any flat pose estimation into equal parts and simulates a "real" multiple animal pose estimation. Note however, that this has some prerequisites. This functionality will split any skeleton into equally sized sub_skeletons with a number of bodyparts equal to the original number of body parts divided by the number given in ANIMALS_NUMBER. It cannot work with skeletons that cannot be evenly split and will throw an error.

By setting SPLIT_MA to TRUE, you can use any "real" multiple animal experiment (including the example).

By setting SPLIT_MA to FALSE, you can use any multiple animal experiment that expect the body parts to appear in style (nose1, neck1, ..., nose2, neck2 ,...). All original triggers can be used by explicit naming of body parts, so that "single" animal experiments can still be utilized in that way.

Please make sure, that only one of the options is set TRUE.

3. Multiple animal experiments where the identity of the animal is not important for the trigger (e.g. a trigger where any animal can activate it) can utilize either single instance models or multiple instance models (SLEAP, maDLC, DLC):

For flat multiple animal pose estimation it is wise to split flat skeletons by using the SPLIT_MA functionality. DLStream is designed in such a way that with multiple skeletons (instances of animals) experiments can also test seperately for any skeleton if the trigger condition is TRUE. This is possible in two ways:

Default in multiple animal experiments is that experiments are passed all skeletons and then handle the trigger check within themselfs. This can look like this for triggers that utilize all skeletons at once:

class ExampleSocialInteractionExperiment:
...

    def check_skeleton(self, frame, skeletons):
...
            #checking if enough animals were detected
            if len(skeletons) >= self._min_animals:
                for trial in self._trials:
                    # check if social interaction trigger is true
                    result, response = self._trials[trial]['trigger'](skeletons=skeletons)
                    plot_triggers_response(frame, response)
...

Or can be realized by passing each available skeleton seperately and testing wether any or all animals test TRUE:

    def check_skeleton(self, frame, skeletons):
...
            #cicling through all animals
            collected_results = []
            for skeleton in skeletons:
                for trial in self._trials:
                    # check if trigger is true for this animal
                    temp_result, response = self._trials[trial]['trigger'](skeleton=skeleton)
                    collected_results.append(temp_result)
                    plot_triggers_response(frame, response)
            #checking if any temp_result was TRUE
            if any(collected_results):
                    result = True
...

Another way is to use the PASS_SEPARATE option in advanced_settings.ini.

When this is set to TRUE, the experiment receives each skeleton separately. This has different consequences depending on the overall experimental design, but the simplest advantage is that skeletons/animals are tested in a row and any animal can trigger a stimulation. With this set to TRUE, you can basically run any SingleAnimalExperiment with multiple animals.