How to set INSTANCE_SEG / WATERSHED for instances with thin and weak connections

[albertding1113-spec]

Hi!

I find when using default .yaml file to train models for instance segmentation, it's hard to link spine to dendrite because the spine neck is very thin.
Should I set GROWTH_MASK_CHANNELS_THRESH lower as follows?
INSTANCE_SEG:
DATA_CHANNELS: ["F", "C"]
WATERSHED:
SEED_CHANNELS: ["F", "C"]
SEED_CHANNELS_THRESH: ["auto", "auto"]
TOPOGRAPHIC_SURFACE_CHANNEL: "F"
GROWTH_MASK_CHANNELS: ["F"]
GROWTH_MASK_CHANNELS_THRESH: [0.3, "auto"]

Or I should use one more parameters such as Db?
DATA_CHANNELS: ["F", "C", "Db"]
DATA_CHANNEL_WEIGHTS: (1.0, 1.0, 0.5)
WATERSHED:
SEED_CHANNELS: ["F", "C"]
SEED_CHANNELS_THRESH: ["auto", "auto"]
TOPOGRAPHIC_SURFACE_CHANNEL: "F"
GROWTH_MASK_CHANNELS: ["F", "Db"]
GROWTH_MASK_CHANNELS_THRESH: [0.3, "auto"]

Looking forward to your reply

[danifranco]

Hi,
You can try adjusting the values in SEED_CHANNELS_THRESH. Now that value is "auto", which means BiaPy will do a Otsu thresholding to determine a value. I would try to choose something like SEED_CHANNELS_THRESH: [0.2, 0.2] so the seeds capture a the dendrites. Those values will be used to threshold F and C channels, in that order (you have more information of the process here, specially in "From channels to seeds and instances" --> "Watershed-based" --> "1. Seed creation"). You should have the raw model predictions in "results" forlder, in a subfolder called "per_image". If you open a predicted image, for example in Fiji, you will see two channels: F + C. This way you can maybe play with the thresholding tool of Fiji to find a proper value for the seeds.

Regarding the problem in general, I can see from these images than the spines are not touching each other? If that's the case maybe you can just use F channel. If you provide me more information of the problem maybe I can guide you a but better :D

[albertding1113-spec]

Hi. thanks for your reply. Here is my .yaml file and my groundtruth (1.6 GB tif img).
8_unext_v1_training.yaml

I've tried different combinations of SEED_CHANNELS_THRESH of F and C. High THRESH leads to misconnection between spines and dendrite, while low THRESH leads to over-connection among neighbors. I cannot find a balance.

Should I use more groundtruth as training set or training a model with more DATA_CHANNELS?

[danifranco]

What is the resolution of the data? The patch size seems to be a bit large for me. I would try this configuration file:

8_unext_v1_training.yaml

Changes:

• Percentile clipping with DATA.NORMALIZATION.PER_CLIP. Low percentile 0.1, upper percentile 99.8.
• Patch size to (20, 256, 256, 1) instead of (20, 512, 512, 1). Increased the batch size a bit form 4 to 6 as it will fit now.
• More data augmentation to avoid using more data:

    G_BLUR: True
    BRIGHTNESS: True
    BRIGHTNESS_FACTOR: (-0.1, 0.1)
    CONTRAST: True
    CONTRAST_FACTOR: (-0.1, 0.1)
    ELASTIC: True
    AFFINE_MODE: 'reflect'

• I remove onecycle lr scheduler as for me I like more the results with warmupcosine. To use that I decreased the learning rate an order of magnitude: 1.E-4.

Which version of BiaPy are you using? If you have downloaded the code I would use the last commit f13bcf6 until we add a new release.

If you want you can send me also a few slices of data so I can check them. If F + C still doesnt work well you can maybe move to F + P with P channel using a skeleton. This is done, in case you want to try, by putting this:

    INSTANCE_SEG:
        DATA_CHANNELS: ['F', 'P']
        DATA_CHANNEL_WEIGHTS: [1.0, 2.0]
        DATA_CHANNELS_EXTRA_OPTS: [{'P': {'type': 'skeleton', 'dilation': 3}}]

You can dilate the skeleton based on the data, here I put 3 because visually seems to be a good number to pick up the spines, but hard to tell without looking at the data.

Hope all this helps!!

[albertding1113-spec]

Hi, thanks for sharing the skills for training neural networks. I share part of my raw data and labels here: https://github.com/albertding1113-spec/TEEX-data-share
I update biapy from 3.6.7 to 3.6.8 and use the last commit today. I'm training new network with your .yaml file now. Any improvements I will share in time.

[danifranco]

I've checked a few images and they look beautiful! The patch size seems to be better at (20, 256, 256, 1) than (20, 512, 512, 1) I think. I see that most of the spines are small compared to a big instance in the bottom-right corner:

I'm not biologist so I don't know what they are, maybe it's the same kind of object at other scale, but from a computer vision perspective the model would need to recognize both structures at the same time and that might be hard. Maybe you can consider separating both types of instances? I think it could be done easily based on size.

Regarding the data channels to use, I would suggest trying F + P as said before, I think that would help a lot in your case. Now, looking at the images, maybe I would dilate 1 or 2 the skeletons instead of 3:

    INSTANCE_SEG:
        DATA_CHANNELS: ['F', 'P']
        DATA_CHANNEL_WEIGHTS: [1.0, 2.0]
        DATA_CHANNELS_EXTRA_OPTS: [{'P': {'type': 'skeleton', 'dilation': 1}}]

You can try different values to see how it changes. The data to train the model, based on the channels selected, is created in a folder near to you ground truth data, so I'd recommend you to open them and see the binary masks the model will try to learn (in the log you will have a print saying where these images are stored exactly). If you don't find them, you can share the log and I could point out the path for you. Remember that these channels are binary, 0/1 values, so if you open them in Fiji you would need to apply a LUT (like glasbey_on_dark) to see them ;)

Another thing you can try is filtering those samples not containing labels, e.g. pure background patches, so to ensure every patch fed into the model will have some information. This normally helps the training when the labels are sparse. You can do it like that (more info here):

DATA:
    TRAIN:
        FILTER_SAMPLES:
          ENABLE: True
          PROPS: [['foreground']]
          VALUES: [[0.0000001]]
          SIGNS: [["lt"]]

Hope it helps! :D

[albertding1113-spec]

Hi, Daniel. I'm gald that the model increases a lot after using your configuration file. The IoU of C channel increases from 0.30 to 0.36, F channel increase from 0.77 to 0.81.
And I also change the feature map to [32, 61, 128, 256, 320]. The IoU of C channel increases to 0.45.

There are still mismatchings between thin and wide instances. Overall, it is much better than before.

Now I' trying with F+P strategies. I still have some questions.

1. What's the ideal value of C channel IoU? Is 0.45 still low for segmentation
2. In my dataset, there exist both thin&weak instances and strong&big instances. Should I continue to increase feature map to imporve the network learning ability for distinguishing these instances.
3. I notice there a new network in source code named unext_v2, but not mentioned in documentation. Does it powerful than v1?
   BiaPy is a powerful platform and I am recommending this project among my labotary and academy colleagues. Thank you very much!

[danifranco]

Hi,

I'm very glad to hear that you have improved the results!

While you are training a F + P model I'd try to adjust a bit the thresholds on the F + C approach. That's why I was suggesting you to open a test image of the "per_image" folder and try to play with the threshold tool of Fiji to adjust these values (we can have a small chat also if you want and I can quickly show you that).

Going through you questions:
1- You can try with a lower value to ensure the instances wil be separated, like 0.3 or 0.2. But it's hard to tell without loooking the predictions. The best is what I suggested before using Fiji.
2- You can try it, yes, but you might need to decrease the batch size if you face a OOM error due to GPU memory.
3- unext_v1 is based on the paper: A ConvNet for the 2020s. unext_v2 is based on the 2nd version of the ConvNeXt family: ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders. It should work better than the v1. What do you mean it is not mentioned in the documentation? It is actually described there as the unext_v1. I'd like to know your feedback on this so I can improve the documentantion :D

Thank you so much for your words! I'm glad you like BiaPy!!

[albertding1113-spec]

Hi Daniel. Thank you very much for your detailed response!
I actually played Fiji threshold tool. And I tried different values of WATERSHED.SEED_CHANNELS_THRESH of F and C channels, from 0.1 to 0.4. But like I said before, I cannot find a balance between misconnection and over-connection of instances of various shapes. Currently I still use [0.2, 0.2] as SEED_CHANNELS_THRESH.

I shared part of per_image here: https://github.com/[albertding1113-spec/TEEX-data-share](https://github.com/albertding1113-spec/TEEX-data-share). If you have time, I would really appreciate it if you could take a look.
As for documention, I selected models from here:
https://biapy.readthedocs.io/en/latest/workflows/instance_segmentation.html

And I didn't find a model named UNeXt-V2 until I read the source code.
Thank you again for your help!
Best regards

[danifranco]

Hi,

I see the problem: the C is not well defined in the dendrites, cause they are small and the connection with the spine is thin, so it's hard for the model to find a good value. I'd suggest going for the F + P with skeletons. I think that will help a lot in this scenario. I'm quite interested in the results you'll get!

Regarding the documentation, thank you so much for pointing that out! I think we need to update the advance parameters of each of the workflows, as each one has some special properties described in config.py that should be documented (@iarganda).