Advice on backbones

[GettyScience2]

Hello!

I am looking for any suggestions/information on how best to pick a backbone for my project. The setup is a resident intruder assay with fish and it is a side-view as that is how normal scoring works in this model. Training appears to be going okay but I have read here that side-view can be difficult and maybe is better with a different backbone, but am struggling to find any information on them to help me make a good decision. Currently using a top-down model as that was suggested over on sleap-nn.

We plan to fix a few issues in the coming weeks (these are currently older videos taken long before we thought we would do this project) such as the reflection on the sides, the distance of the camera from the tank. For now I am just trying to figure out the best model for this view and species.

Beyond the backbone, any other configuration suggestions are appreciated.

Any suggestions?

[gitttt-1234]

Hi @GettyScience2 !

Since your fish is small relative to the frame, top-down would be a better choice. Ref: model types doc: the centroid stage finds the fish, then crops in to give the pose model much more effective image to work with.

On backbones:

sleap-nn ships three "native" backbones, plus a newer option to reuse pretrained HuggingFace encoders: UNet, ConvNeXt, SwinT. Given your setup (side-view, small/distant subject, and limited labeled frames so far), I'd actually lead with a pretrained backbone rather than UNet: ConvNeXt (tiny, ImageNet-pretrained). However, ConvNext models are bigger than UNets, if you want to experiment with UNet, try setting filters to 32 or 64.

Other config knobs worth tuning for your case:

• Anchor part: pick a specific, reliably-visible node (e.g. a mid-body point) as the centroid anchor rather than the bounding-box center (the default when no anchor is set).
• Receptive field / max_stride: rule of thumb is receptive field is around size of your animal. The centroid model uses a larger receptive field/coarser resolution (it just needs to find the fish) with input scaling of 0.5, keep increasing max stride until it fits an entire instance (you can check the receptive field in the GUI training dialog window).
• Sigma (confidence map spread): a larger sigma (~5) is easier to learn and a reasonable starting point for keypoints that are occasionally occluded or low-contrast against the background. You can also tighten it later (e.g. to ~2.5) once training is going well.
• Flip augmentation might help too, just note you'll need symmetric node pairs set in your skeleton for left/right flips to swap keypoints correctly.
• Online Hard Keypoint Mining (trainer_config.online_hard_keypoint_mining): upweights consistently-difficult keypoints during training. You could try turning this on a second pass of training if you notice a couple of nodes underperforming relative to the rest.

Our recommendation is the same iterative loop: train, look at where it's failing (is it the occluded keypoints? the glare? small size?), and adjust one thing at a time. If the fish end up huddling together often, bottom-up models tend to handle overlapping/touching animals better than top-down.

I'd also recommend trying our Config Picker app, which suggests a good set of default config parameters based on your data.

Let us know if you have any questions!

Thanks,

Divya

[GettyScience2]

Hi! Thank you for your detailed response! This information makes things a lot clearer for me.