Trouble reproducing ground truth results

[erthher]

Hi,

I am trying to use the evaluation script to reproduce the ground truth results.
The label has a shape of (1024, 2048, 3) but the segmentation result has a shape of (1024, 2048). As a result the fast_hist function throws the following error:
IndexError: index 2097152 is out of bounds for axis 1 with size 2097152.

If I try to select only one of the label channels or use np.repeat to stack the segmentation result with itself I get very poor results for the ground truth (Mean Pixel accuracy < 0.04).

Is that the intended behavior of the script?

Thanks in advance.

[junyanz]

@tinghuiz

[zhuolong3]

I met the same problem too. And I am so confused about it.I have got this. But I can't get the right metrics....@junyanz Could I get some suggestions?

[tinghuiz]

Hi, please refer to the notes under "Evaluating Labels2Photos on Cityscapes" in the README.

[erthher]

@tinghuiz Thank you for your reply.

I'm afraid these notes aren't related to the problem at hand since the problem isn't in the dimensions of the images but in the number of channels.

Just to be sure I uncommented line 46 but I still got the same error.

[erthher]

If anyone else encountered this problem - this is due to evaluating the test set and not the train or validation set. In the Cityscapes dataset the test set doesn't have the same label format and meaning as in the other splits.

[zhuolong3]

@erthher Excuse me . I am so curious about the solution of the problem ? Could I get your email and ask some questions ,plz ?

[KaitiSt]

If anyone else encountered this problem - this is due to evaluating the test set and not the train or validation set. In the Cityscapes dataset the test set doesn't have the same label format and meaning as in the other splits.

Hello @erthher , @tinghuiz
I am facing the same issue here . Did you manage to proceed with the evaluation of the testing set ? If yes, can you please explain how ?

[tinghuiz]

@KaitiSt As mentioned in Section 6.2 "Training details" of the paper, we used the Cityscapes validation set for testing.

[KaitiSt]

Hello @tinghuiz ,
Thank you for your response. I updated the cityscape directory with the original Cityscape validation set from the Cityscape website , which includes images of size 1024 x 2048 . And, also, I updated the result directory to have my predicted images , which are size 256 x 256 .

I tried different scenarios in order to make the evaluation work .

Based on the evaluation code the predicted images will be resized from 256 x 256 to the label size which is 1024 x 2048 and push them into the segmentation network. If I keep the code as it is i am getting the below exception :

F0415 12:42:11.680351 10204 syncedmem.cpp:71] Check failed: error == cudaSuccess (2 vs. 0) out of memory

Based on your comment , I can understand that the pre trained model does not work well with 1024 x 2048. So if i remove the resize of the input image to 1024 x 2048 , the segmentation is done , but it's failing in the fast_hist method with the below exception:

IndexError: index 65553 is out of bounds for axis 0 with size 65536

But, as per your your comment once more , if I resize the original cityscape image - label to 256 x 256 the evaluation is working.

The results though are not so good

Example:
0_input (256x256)

0_pred (256x256)

0_gt (256x256)

[sigtot]

@KaitiSt did you ever solve this last problem? We're facing the same issue with the entire image being marked as road (purple).