Multiclass segmentation

[JaledMC]

Hello @karolzak ,

Thanks for this great repo. In Customizable U-Net, it seems like multiclass segmentation can be done. But, what is the proper dataset format? With one hot encoding, one ground mask image per class for each example is needed. There is another way, assign each pixel its class (1, 2, 3, ...). But you use normalization to force label values between 0 and 1.

Could you provide some insight about this, please?

[JordanMakesMaps]

@JaledMC could you point me to where you see the labels being normalized between 0 and 1?

[JaledMC]

Hello @JordanMakesMaps

In the notebooks (thank to @karolzak for these useful scripts), you can see all steps needed for data preprocessing and training. If you go to Get data into correct shape, dtype and range (0.0-1.0), this two lines do normalization in both images (x), and labeled annotations (y):

x = np.asarray(imgs_np, dtype=np.float32)/255
y = np.asarray(masks_np, dtype=np.float32)/255

In this way, pixel values go from 0 to 1 (max), and assign each pixel with its class (1, 2, 3 ...) can't be done. One solution could be use one hot encoding, but I don't know the filenames format for each mask.

[JordanMakesMaps]

@JaledMC thanks, I forgot about the notebooks. Yeah I'm not sure about that, but I feel like it was just an error from copy and pasting code? I'm using the network, normalizing the images ([0, 1]), but one-hot-encoding the masks the same way I do with other architectures.

This implementation works pretty good compared to others.

[karolzak]

Hi @JaledMC and @JordanMakesMaps
Thanks for your interest in this package :).
I think you brought up a good topic for discussion.
The way I build models for multiple classes is basically training separate model per class, so in fact I divide the multiclass segmentation problem into multiple binary segmentation problems. That's what I found working quite well in my projects.

It might be a good idea to prepare an example for multiclass segmentation as well.
The way I implemented custom_unet right now allows to build multiclass model ->

keras-unet/keras_unet/models/custom_unet.py

Line 72 in 531f49f

outputs = Conv2D(num_classes, (1, 1), activation=output_activation) (x)

It's just a matter of changing num_classes argument and you would need to shape your mask in a different way (layer per class??), so for multiclass segmentation you would need a mask of shape (width, height, num_classes).

Let me know what you think and if that makes sense to you.
Thanks!

[karolzak]

Closing for now since there no activity happening for 2 weeks

[JordanMakesMaps]

@karolzak, so you train multiple models individually, one for each class? When you perform predictions on images with multiple classes present, do you just save the prediction from each model and combine them overall?

Can you load multiple models into memory at once?

[karolzak]

so you train multiple models individually, one for each class? When you perform predictions on images with multiple classes present, do you just save the prediction from each model and combine them overall?

@JordanMakesMaps , yes, that's more or less how I'm doing it.

Can you load multiple models into memory at once?

Yes you can. The problem with keras is that by default it holds a global session, so when you're working with multiple models at once you need to make sure that you're using separate sessions and models on different graphs.
This thread talks about it in more detail: keras-team/keras#8538 (comment)