3D segmentation using a set of 2d segmentations

[Fafa87]

I have a 3d single channel volume that I would like to segment.
If I run 3d version (use_3D=True) then the results are very noise - with complies with the statements in the docs.

However I do not know how to use the workaround (run 2D and stitch). If I run this:

model_cyto2 = cellpose.models.Cellpose(gpu=False, model_type='cyto2')
imagery_subset_cyto2, _, _, _ = model_cyto2.eval(sample_3d diameter=100, channels=[0,0], do_3D=False, stitch_threshold=0.1)

Then I get a single plane.

If I go with converting 3d volume into list:

model_cyto2 = cellpose.models.Cellpose(gpu=False, model_type='cyto2')
imagery_subset_cyto2, _, _, _ = model_cyto2.eval(list(sample_3d) diameter=100, channels=[0,0], do_3D=False, stitch_threshold=0.1)

There is no stitching done :/
What do I miss?

My workaround is to manually run:
cellpose_utils.stitch3D(imagery_subset_cyto2, stitch_threshold=0.1)
However it fails if there are empty slides:

    348         iou = metrics._intersection_over_union(masks[i+1], masks[i])[1:,1:]
    349         iou[iou < stitch_threshold] = 0.0
--> 350         iou[iou < iou.max(axis=0)] = 0.0
    351         istitch = iou.argmax(axis=1) + 1
    352         ino = np.nonzero(iou.max(axis=1)==0.0)[0] 
ValueError: zero-size array to reduction operation maximum which has no identity

Obviously I can manually go through the slices and run stitch only in the ranges where there are any cells but... :)

Btw. I really love the quality of the segmentation of CellPose - great job!

[chrisroat]

What about using both do_3d=True and stitch_threshold=0.1?

[puisheelee]

I ran into the same problem some time ago. I made some changes to the stitch3D function to take care of that (though maybe this is not the most elegant way of solving this problem). The first if is to deal with empty first z slice, and the second is for empty z slices in the middle of the volume and making sure the numbering of cells doesn't restart from 1.

def stitch3D(masks, stitch_threshold=0.25):
    """ stitch 2D masks into 3D volume with stitch_threshold on IOU """
    
    mmax = masks[0].max()
    empty = 0
    
    for i in range(len(masks)-1):
        iou = metrics._intersection_over_union(masks[i+1], masks[i])[1:,1:]
        if not iou.size and empty == 0:
            masks[i+1] = masks[i+1]
            mmax = masks[i+1].max()
        elif not iou.size and not empty == 0:
            icount = masks[i+1].max()
            istitch = np.arange(mmax+1, mmax + icount+1, 1, int)
            mmax += icount
            istitch = np.append(np.array(0), istitch)
            masks[i+1] = istitch[masks[i+1]]
        else:
            iou[iou < stitch_threshold] = 0.0
            iou[iou < iou.max(axis=0)] = 0.0
            istitch = iou.argmax(axis=1) + 1
            ino = np.nonzero(iou.max(axis=1)==0.0)[0]
            istitch[ino] = np.arange(mmax+1, mmax+len(ino)+1, 1, int)
            mmax += len(ino)
            istitch = np.append(np.array(0), istitch)
            masks[i+1] = istitch[masks[i+1]]
            empty = 1
            
    return masks

[vineet-kureel]

Hi all,
I am trying to implement the 2D segmentation workaround for 3D images. I have few questions, 1. how you evaluate whether stitching worked or not. I assumed if it works then all the labels should have a unique id but in my case, the program is running fine but the labels have different id per image. I suspect the stitching is not happening.
From the thread, I understand the stitching function fails if there is an empty image in the stack but in my case, all images in a stack have segmented cells.

[puisheelee]

Hi.
What is the stitching threshold that you used? Maybe try lower the threshold? How do you perform the stitching?

What you could do is to run the segmentation on one (or a few) test image stack(s) for various threshold values, and do a quick check with GUI. Also, to make sure no two cells share the same id, you could then separate the pixels belonging to individual cells and check if all cells have its unique id.

[vineet-kureel]

Hi @puisheelee,
I am using the default stitching value of 0.1, I did try playing around with the threshold. I am using the jupyter notebook for cellpose. When I run the stack the notebook generates the output stack with each frame having disconnected (I mean different label id for each segmentation) labels, despite having "do_3d = False".
The Output shows the individual plane being segmented but no report of stitching.

I am going to try the steps you told me in GUI, Wish me luck :)
Thank you for your guidance.

[carsen-stringer]

thanks @puisheelee for the help here, do you want to open a pull request with your code so we can incorporate it into cellpose?