Problem on Reproducing Scannet Segmentation Results

[hasanari]

Hi,

I tried to reproduce the results for Scannet Dataset on Segmentation Task but I cannot reproduce the mentioned results. The accuracy just around 77.76% (Validation) and ~85% for training data.
I used the "pointnet++ preprocessed data" with provided scannet-seg hyper-parameters.

Can you give me a clue what I should do to reproduce the result? And also how long do you train the model to achieve the published result?

[MingChaoSun]

Hi，

Our code on Github has updated many times. The network structure, hyper-parameters and data preprocessing has been changed. By the way, due to the Scannet dataset is very easy to overfit (contains some messy annotations)，some strategies that have improved significantly on other datasets but do not work here. Especially，We rewrite the data preprocessing code (cut the point cloud into blocks) for Scannet, S3DIS and Semantic3D datasets which contain large scale scenes. The high speed new data preprocessing method makes cut the mass points (Semantic3D dataset) possible, at the same time, improves the segmentation performance on S3DIS dataset. However, the new data preprocessing strategie cause performance degradation on Scannet segmentation task(about 1~2%).

Even using new data, the accuracy should not be less than 83% (Validation). I will use the code and hyper-parameters train again to make sure of this. In addition, we share our tensorboard summary file (events....), maybe you can find some problems through tensorboard curves.

[hasanari]

I see on your tensorboard summary events, you already get 83% on first few iterations which is strange, because mine was only at 75%, all validations. Are you using any pretrained model? Or you train them from scratch?

NB: Thanks for retraining, I will wait for your result.

[MingChaoSun]

Hi, hasanari

I cloned the last version then followed the README to prepare data and retrain. After about 60K iterations，the accuracy(validations) reached about 83.6%. BTW， We don‘t use any pretrained model and train from scratch.

I think this version should be fine. Maybe you can check your pipeline again？ I have no idea at the moment, because we haven't had a similar situation before. If you have other questions, please feel free to contact us～

[hasanari]

Hi Again,

Thanks for the update, I retrained them again but I can only get 82.6% (Validation) tensorboard summary file.
The only different on the hyper parameter are:
batch_size = 8 step_val = 4000
I can't train more batch because it will be OOM. My question is that is smaller batches hurt performance that much or is there something that I need to fix?

And I also try it with 3sdis dataset, train with "1, 2, 3, 4, 6" Area, Test with Area 5, and I get MIoU 95.64% for test data, is it normal? Because it's much lower on your paper.

[burui11087]

@hasanari

I want to know setting details of S3DIS, by the way, I think you need to double check there is no Area 5 splited into train set. As far as I am concerned, 95.64% maybe is IoU value of "ceiling".

If there is no problem metioned above, I am very glad to know how to get 95.64%.

Thanks!

[MingChaoSun]

Hi，hasanari

The small batchsize can cause performance degradation. I will train again with batchsize 8 and compare with your results～

[MingChaoSun]

Hi, hasanari

With batch_size = 8 and after 200K iterations, I get about 82.9% (Validation). So the performance degradation seems to be caused by the small batchsize.