*Segmentation is performed using the cellpose convolutional neural network published in Nature (**citation)**.* 

To improve performance user can retrain cellpose to fit its data. To segment cytoplasm auto-fluorescence from fish signal is sufficient most of the time. To avoid long computational time it is highly recommended to set up a GPU event a "weak" one espacially for 3D segmentation.

# 1. How Small Fish uses Cellpose for segmentation

## 2D segmentation

Two masks, **nuclei** and **cytoplasm** are predicted using cellpose cpsam model and expected size of object in pixel (a rough value will perform well). For 3D stacked images predictions can be performed in 2D on either Z-mean projection, z-max projection or single slice projection OR in 3D.  

For more information on cellpose cpsam model, here is the quote to the biorxiv paper :  
*Cellpose-SAM: superhuman generalization for cellular segmentation Marius Pachitariu, Michael Rariden, Carsen Stringer bioRxiv 2025.04.28.651001; doi: https://doi.org/10.1101/2025.04.28.651001*

Given that [detection](https://github.com/SmallFishGUI/small_fish_gui/wiki/Detection/) is performed in **3D**, segmentation will create a bias on measurements regarding classification of nuclear spots as well as distances to nucleus/cytoplasm borders. Depending on the measure software implementation can either extend 2D masks uniformly along Z axis (classification of nuclear spots) or project spots in 2 dimensions (distance features). For details on how measurements are computed refer to [Measurements section](https://github.com/SmallFishGUI/small_fish_gui/wiki/Measurement-&-Quantification).  
Since SmallFish 2.0 segmentation can be performed in 3D, however all features computed through the fish-quant framework were engineered for 2D segmentation as a consequence segmentation is projected back into 2D. However for cell volume and individually spot extraction quantification is performed in 3D allowing to discriminates single molecules below/above nuclei from single molecules inside nuclei.

### Details of interaction with cellpose

The new cellpose model doesn't rely anymore on several channels to make its prediction and as such when retraining the model the user needs no more to concern with an optional dapi channel. What is more while using cellpose you will see printed in the application terminal "object size is depreciated" but from my experience it is not. Cellpose is trained using images where cells size average around 30px consequently when segmenting an image where cells are bigger a resampling is nessecary. I found this parameter to be of critical importance in fact when segmentation underperforms I recommend trying different sizes before proceeding to retraining model.

Additionaly, it is worth mentioning that Small Fish will always attemp to segment nuclei whereas cytoplasm is optional thanks to the option 'segment only nuclei'. However if the user doesn't want to segment nuclei and wants to **use a cellpose model** that was trained to predict cytoplasm masks **without the nucleus channel** it is possible by passing the cytoplasm to the software as if it was the nuclei signal.

To pass both nuclei and cytoplasm images to the software you can use multi-channel images or open two images during segmentation process.

### 'Segment only nuclei' option

For users who would like to quantify activity in the nuclei without bothering achieving cytoplasm segmentation the *'segment only nuclei'* option can be used. In this case nuclei segmentation will be performed as mentionned above and resulting mask will be used both for the nuclei boundaries and the cell boundaries.

## Cytoplasm/Nuclei matching

After segmenting separetly nuclei and cytoplasm, Small Fish runs a step of matching so that nuclei overlapping with a cytoplasm share the same label value. During this process if a cytoplasm doesn't overlap with any nuclei it is discarded, equivalently, a nucleus without cytoplasm is discarded. Finally if a cytoplasm finds more than one nucleus in its region the nucleus that biggest overlapping region will be kept and other discarded.  

One should keep in mind that segmented cells that lie on the border of the field of view are not discarded at the moment of segmentation but **will be later on** during quantification as they are considered incomplete.

*Note : When segmentation is performing poorly (i.e all lot of cells are missed) try to run with the option **'segment only nuclei'** first only on your nuclei marker channel and then on your cytoplasm marker channel to assess if most cells are not discarded during matching.*