Problem on semantic segmentation evaluation

[rainfall1998]

Hi, I have evaluated the predicted semantic of Scannet_0050_00. I think this sequece has worse segmentation result than others based on semantic picture. However, the IoU_f, IoU_w, IoU_m are 0.73332 , 0.76606, 0.78970, is much higher than table 3 in paper about 0.62 0.52 0.57.

The GT used is based on the scannet semantic label, while floor contains 1, 161, 52; wall contains 3,140. The evaluation code is based on semantic_nerf calculate_segmentation_metrics, with ignore_label=-1.

Is there anything wrong? If possible, could I get your calculation formula and evaluation code for semantic evaluation?

Thanks for your work and looking forward to your reply!

[ghy0324]

Hi! Thanks for your interest!

We use the nyu40 id (you can map the id using the label mapping file named scannetv2-labels.combined.tsv provided by ScanNet). We calculate iou (averaged on each frame) for each scene and average again on each scene as the final results. And we believe that our method can improve the semantic results significantly no matter in which evaluation manner.

[rainfall1998]

Sorry, I may not have made the problem clear. Because of time, I only tested the 0050 sequence. Judging from the semantic segmentation picture, 0050 sequence should have a lower IOU than other sequences, due to the incorrect segmentation of the chair. But the evaluation result is still higher than table 3 in paper. The IoU_f, IoU_w, IoU_m are 0.73332 , 0.76606, 0.78970, compared to table 3 in paper about 0.62 0.52 0.57.

So I wonder if the method I used for evaluation has some problem. Or maybe the groundtruth value I generated is not correct ? Since I hadn't done work on semantic before, I didn't konw what was wrong with my evaluation method. The problem I meet is that the result is much higher than those given in the article, so it feels like something might be wrong.