Segmentation fault at Softmax

[codecolony]

Hi Alex,

Hope someone can help me with this issue.
I'm just reusing the network architecture with only 4 output classes. Image dimensions remain same. Made the necessary changes in the prototext files. I get the following error when I start the training.

I0531 09:45:15.546435 2031686400 net.cpp:248] Memory required for data: 1068595224
I0531 09:45:15.546778 2031686400 solver.cpp:42] Solver scaffolding done.
I0531 09:45:15.546954 2031686400 solver.cpp:250] Solving VGG_ILSVRC_16_layer
I0531 09:45:15.546959 2031686400 solver.cpp:251] Learning Rate Policy: step
*** Aborted at 1464668121 (unix time) try "date -d @1464668121" if you are using GNU date ***
PC: @ 0x10b25b442 caffe::SoftmaxWithLossLayer<>::Forward_cpu()
*** SIGSEGV (@0x216caa000) received by PID 55425 (TID 0x7fff79191300) stack trace: ***
@ 0x7fff918dff1a _sigtramp
@ 0x100000000000000 (unknown)
@ 0x10b2209d9 caffe::Layer<>::Forward()
@ 0x10b2760d9 caffe::Net<>::ForwardFromTo()
@ 0x10b276718 caffe::Net<>::Forward()
@ 0x10b285dcf caffe::Solver<>::Step()
@ 0x10b28577c caffe::Solver<>::Solve()
@ 0x10b1b1594 train()
@ 0x10b1b392f main
@ 0x7fff8c7e25c9 start
@ 0x4 (unknown)
Segmentation fault: 11

Any help in this regard. Appreciate your time being spent on this.

[alexgkendall]

Typically I observe these kind of errors when there is an invalid label. ie. if you are outputting four classes then a pixel which is labelled > 3? do you only observe this in cpu mode?

[codecolony]

Hi Alex,

Thanks for the quick response.
Yes, I have only CPU option to work on.

The fourth class is void (black) and as you noted I hardly have any instances where it is marked (except for couple of images with very less pixels). Does this affect my training and cause this error somehow?

[alexgkendall]

The pixels in your label images should be between 0 and 3 inclusive.

[codecolony]

Hi Alex,

Ok, looks like I made some silly mistake somewhere. I'm bit confused after reading your above reply. Where exactly do we label the pixels with 0-3 values? I have used the Interactive Labeler tool found here to paint the regions I'm interested in and the results look like this. Nowhere did I have to do any sort of color labeling. Did I do something wrong here?

I could successfully train using camvid dataset though. I get this error only when I switch to my dataset. The error is seen when I issue caffe train command.

The following are the changes I did in my train.prototxt

layer {
bottom: "conv1_2_D"
top: "conv1_1_D"
name: "conv1_1_D"
type: "Convolution"
param {
lr_mult: 1
decay_mult: 1
}
param {
lr_mult: 2
decay_mult: 0
}
convolution_param {
weight_filler {
type: "msra"
}
bias_filler {
type: "constant"
}
num_output: 4
pad: 1
kernel_size: 3
}
}
layer {
name: "loss"
type: "SoftmaxWithLoss"
bottom: "conv1_1_D"
bottom: "label"
top: "loss"
softmax_param {engine: CAFFE}
#loss_param: {
#weight_by_label_freqs: true
#ignore_label: 4
#class_weighting: 0.2595
#class_weighting: 0.1826
#class_weighting: 4.5640
#class_weighting: 0.1417
#class_weighting: 0.9051
#class_weighting: 0.3826
#class_weighting: 9.6446
#class_weighting: 1.8418
#class_weighting: 0.6823
#class_weighting: 6.2478
#class_weighting: 7.3614
#}
}

Please let me know what else I missed to start the training.

[alexgkendall]

Hey - all looks good except the labels should be grayscale, with label values (in your case 0,1,2 or 3) in each pixel. See the camvid data as an example.

[codecolony]

Oh! Thanks for the clarification. Appreciate your patience with all my novice queries.

I had read about the single channel ground truth images but had not quite understood it. So, if I understood correctly, I will convert my images to grayscale and assign each color a fixed value (0,1,2 or 3) which are basically intensity values, right?

[alexgkendall]

correct :)

[jonVolta]

Hi Alex,
i am doing my final year project on semantic face segmentation using SegNet. And i followed the segnet tutorial and it works good, but when i tried to train the network with my own dataset i got a 'segmentation fault (core dumped)'. I couldn't understand where i made a mistake.
I have a face image dataset and it's ground truth image, and my task is to segment face,hair,eye,nose,mouth and background total of six labels; what i did was as follow:-
1, convert the ground truth image in to gray scale image.
2, i assign a label from 0 to 5
3, create the lmdb file for both data and the ground truth image.
4, i make all the necessary changes on train.prototxt file.
please i really need your help Alex.
i attached the .zip folder of my code below
attachedFolder.zip