Accuracy of the model

[louislai]

Hi I was just wondering if you can publicise the accuracy of the model trained on your setup (in term of class IOU and category IOU). I was trying out your caffe implementation but seem like the results I got were not as good as what were described in the paper.

[TimoSaemann]

Hi,
The following results are for the validation set (with the cityscapes_weights.caffemodel). I resized the images from 2048x1024 px to 1024x512 px and after the segmentation process I used bilinear interpolation to upsample the images again to 2048x1024 px.
Surprisingly, the classes bus and train were not learned. Perhaps the class weighting was too low for this classes.
The category IOU is similar to the one mentioned in the paper (78,6 vs 80,4). Of course, these numbers are not direct comparable, since the numbers in the paper are from the test set and not from the validation set.
Anyway, I want to mention that I trained it only with one P100 (16 GB) and not with four Titan X (12 GB * 4) as described in the paper. For this reason I had to take a much smaller batch size, which leads in general to slightly worse results.

classes IoU

road : 0.949
sidewalk : 0.680
building : 0.824
wall : 0.294
fence : 0.395
pole : 0.459
traffic light : 0.252
traffic sign : 0.431
vegetation : 0.841
terrain : 0.408
sky : 0.855
person : 0.674
rider : 0.361
car : 0.868
truck : 0.198
bus : 0.001
train : 0.000
motorcycle : 0.265
bicycle : 0.553

Score Average : 0.490

categories IoU

flat : 0.967
nature : 0.843
object : 0.473
sky : 0.855
construction : 0.832
human : 0.681
vehicle : 0.847

Score Average : 0.786

[louislai]

Hi, yes I realized that resizing the input images to 1024x512, then scaling up the prediction via interpolation would give comparable results. Initially, I ran the model on un-resized 2048x1024 input images and get the output segmentation directly, which was why I got very bad accuracy.