multi class segmentation

[masaladose]

Hi,
This is probably the best CNTK image segmentation example I have seen so far. Thank you!
A few comments:

1. shouldn't the conv9 be: "Convolution((3,3), 32, init=glorot_uniform(), activation=relu, pad=True)(up9)"?Instead of "Convolution((3,3), 64, init=glorot_uniform(), activation=relu, pad=True)(up9)"

2. I am trying to modify this to do multi-class segmentation. I modified conv10 as:
   conv10 = Convolution((1,1), num_class, init=glorot_uniform(), activation=sigmoid, pad=True)(conv9)
   and made the target mask also multi class. Using your example, one class is circles another is square.
   However, I am unable to get segmentation output for both classes. It's usually one class that is segmented and the other is all blank.

I was wondering if you had a multi class segmentation implementation.

Thanks

[usuyama]

1. Seems you're right.
2. That should work. Could you share your code so that I can reproduce the issue?

[masaladose]

def create_model(input, nLabels): conv1 = Convolution((3,3), 32, init=glorot_uniform(), activation=relu, pad=True)(input) conv1 = Convolution((3,3), 32, init=glorot_uniform(), activation=relu, pad=True)(conv1) pool1 = MaxPooling((2,2), strides=(2,2))(conv1) conv2 = Convolution((3,3), 64, init=glorot_uniform(), activation=relu, pad=True)(pool1) conv2 = Convolution((3,3), 64, init=glorot_uniform(), activation=relu, pad=True)(conv2) pool2 = MaxPooling((2,2), strides=(2,2))(conv2) conv3 = Convolution((3,3), 128, init=glorot_uniform(), activation=relu, pad=True)(pool2) conv3 = Convolution((3,3), 128, init=glorot_uniform(), activation=relu, pad=True)(conv3) pool3 = MaxPooling((2,2), strides=(2,2))(conv3) conv4 = Convolution((3,3), 256, init=glorot_uniform(), activation=relu, pad=True)(pool3) conv4 = Convolution((3,3), 256, init=glorot_uniform(), activation=relu, pad=True)(conv4) pool4 = MaxPooling((2,2), strides=(2,2))(conv4) conv5 = Convolution((3,3), 512, init=glorot_uniform(), activation=relu, pad=True)(pool4) conv5 = Convolution((3,3), 512, init=glorot_uniform(), activation=relu, pad=True)(conv5) up6 = C.splice(UpSampling2D(conv5), conv4, axis=0) conv6 = Convolution((3,3), 256, init=glorot_uniform(), activation=relu, pad=True)(up6) conv6 = Convolution((3,3), 256, init=glorot_uniform(), activation=relu, pad=True)(conv6) up7 = C.splice(UpSampling2D(conv6), conv3, axis=0) conv7 = Convolution((3,3), 128, init=glorot_uniform(), activation=relu, pad=True)(up7) conv7 = Convolution((3,3), 128, init=glorot_uniform(), activation=relu, pad=True)(conv7) up8 = C.splice(UpSampling2D(conv7), conv2, axis=0) conv8 = Convolution((3,3), 64, init=glorot_uniform(), activation=relu, pad=True)(up8) conv8 = Convolution((3,3), 64, init=glorot_uniform(), activation=relu, pad=True)(conv8) up9 = C.splice(UpSampling2D(conv8), conv1, axis=0) conv9 = Convolution((3,3), 32, init=glorot_uniform(), activation=relu, pad=True)(up9) conv9 = Convolution((3,3), 32, init=glorot_uniform(), activation=relu, pad=True)(conv9) num_class = nLabels conv10 = Convolution((1,1), num_class, init=glorot_uniform(), activation=sigmoid, pad=True)(conv9) return conv10 def generate_random_data_labels(shape, count, nLabel): x, y = zip(*[generate_img_and_mask_labels((shape[1], shape[2]), nLabel) for i in range(0, count)]) X = np.reshape(np.array(x), (count, 1, shape[1], shape[2])) Y = np.reshape(np.array(y), (count, nLabel, shape[1], shape[2])) return X.astype(np.float32), Y.astype(np.float32) def generate_img_and_mask_labels(shape, nLabel): triangle_location = get_random_location(shape) circle_location1 = get_random_location(shape, zoom=0.7) circle_location2 = get_random_location(shape, zoom=0.7) mesh_location = get_random_location(shape) square_location = get_random_location(shape) plus_location = get_random_location(shape) # Create input image arr = np.zeros(shape, dtype=bool) arr = add_triangle(arr, *triangle_location) arr = add_circle(arr, *circle_location1) arr = add_circle(arr, *circle_location2, fill=True) arr = add_mesh_square(arr, *mesh_location) arr = add_filled_square(arr, *square_location) arr = add_plus(arr, *plus_location) # Create target mask mask = np.zeros(shape, dtype=bool) maskCirc = add_circle(mask, *circle_location1) maskSq = add_filled_square(mask, *square_location) mask = np.vstack([maskSq[np.newaxis, ...], maskCirc[np.newaxis, ...]]) return arr, mask ## These are the commands for training and testing: nLabels=2 input_images, target_masks = simulation.generate_random_data_labels(shape=(1, 128, 128), count=2048, nLabel=2) #input_images = imvalues[0] shape_input = input_images[0].shape shape_target = target_masks[0].shape data_size = input_images.shape[0] # Split data test_portion = int(data_size * 0.1) indices = np.random.permutation(data_size) test_indices = indices[:test_portion] training_indices = indices[test_portion:] test_data = (input_images[test_indices], target_masks[test_indices]) training_data = (input_images[training_indices], target_masks[training_indices]) # Create model x = C.input_variable(shape_input) y = C.input_variable(shape_target) z = cntk_unet.create_model(x, nLabels) #dice_coef = cntk_unet.dice_coefficient(z, y) dice_coef = cntk_unet.dice_coefficient_labels(z, y) #pred = C.softmax(z) #train_loss = cntk_unet.cost_func(z, y) #pe = C.classification_error(z, y) # Load the saved model if specified checkpoint_file = "cntk-unet.dnn" z.load(checkpoint_file) # Prepare model and trainer lr = learning_rate_schedule(1e-5, UnitType.sample) momentum = C.learners.momentum_as_time_constant_schedule(10000) trainer = C.Trainer(z, (-dice_coef, -dice_coef), C.learners.adam(z.parameters, lr=lr, momentum=momentum)) #trainer = C.Trainer(z, (-train_loss, -dice_coef), C.learners.adam(z.parameters, lr=lr, momentum=momentum)) # Get minibatches of training data and perform model training minibatch_size = 30 num_epochs =100 training_errors = [] test_errors = [] for e in range(0, num_epochs): for i in range(0, int(len(training_data[0]) / minibatch_size)): data_x, data_y = slice_minibatch(training_data[0], training_data[1], i, minibatch_size) trainer.train_minibatch({x: data_x, y: data_y}) # Measure training error training_error = measure_error(training_data[0], training_data[1], x, y, trainer, minibatch_size) training_errors.append(training_error) # Measure test error test_error = measure_error(test_data[0], test_data[1], x, y, trainer, minibatch_size) test_errors.append(test_error) print("epoch #{}: training_error={}, test_error={}".format(e, training_errors[-1], test_errors[-1]))

…

On Thu, Aug 3, 2017 at 12:43 AM, Naoto Usuyama ***@***.***> wrote: 1. Seems you're right. 2. That should work. Could you share your code so that I can reproduce? — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUJxwHaJRQWBVnwGnWNiuKvpRSTnb_kkks5sUU_ggaJpZM4Oll0X> .

[usuyama]

Thanks for the code. I think the problem was the dice coefficient, which can be dominated by a class with larger region. I changed the loss function to per-class dice coefficient then it's working now. Please check the latest commit.

[usuyama]

@masaladose FYI - I just posted fully-conv-nets for semantic segmentation, too. Might work better than U-net depending on problems.
https://github.com/usuyama/cntk-fully-convolutional-networks

[masaladose]

Thanks! I will check it out. I have to admit that it's your post that convinced me to switch to CNTK from Caffe and TensorFlow. Things seem much easier and faster here.

…

-Ajay

On Sun, Aug 20, 2017 at 12:36 AM, Naoto Usuyama ***@***.***> wrote: @masaladose <https://github.com/masaladose> FYI - I just posted fully-conv-nets for semantic segmentation, too. Might work better than U-net depending on problems. https://github.com/usuyama/cntk-fully-convolutional-networks — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AUJxwJfnZcBwqlxwCU1nV_4uGa6ASBcEks5sZ7frgaJpZM4Oll0X> .

[usuyama]

I'm so happy to hear that! Let me know if you have ideas for improvements!