Performance on SUN RGB-D

[Sharpiless]

I get some trouble when reproducing the results on the SUN RGB-Ddataset. When I train FCAF3D with the following command for 3 times:

CUDA_VISIBLE_DEVICES=5,6 bash tools/dist_train.sh configs/fcaf3d/fcaf3d_sunrgbd-3d-10class.py 2

I get results as follows:

+-------------+---------+---------+---------+---------+
| classes     | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 |
+-------------+---------+---------+---------+---------+
| bed         | 0.8719  | 0.9845  | 0.6141  | 0.7456  |
| table       | 0.4641  | 0.9672  | 0.2555  | 0.6465  |
| sofa        | 0.6754  | 0.9745  | 0.4773  | 0.7703  |
| chair       | 0.7990  | 0.9602  | 0.6536  | 0.8373  |
| toilet      | 0.9059  | 0.9931  | 0.6496  | 0.8138  |
| desk        | 0.2875  | 0.9431  | 0.0774  | 0.4819  |
| dresser     | 0.3682  | 0.9358  | 0.1909  | 0.5780  |
| night_stand | 0.6696  | 0.9647  | 0.5330  | 0.8118  |
| bookshelf   | 0.2164  | 0.8156  | 0.0430  | 0.2376  |
| bathtub     | 0.7385  | 0.9388  | 0.4736  | 0.6939  |
+-------------+---------+---------+---------+---------+
| Overall     | 0.5997  | 0.9477  | 0.3968  | 0.6617  |
+-------------+---------+---------+---------+---------+

+-------------+---------+---------+---------+---------+
| classes     | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 |
+-------------+---------+---------+---------+---------+
| bed         | 0.8724  | 0.9883  | 0.6265  | 0.7553  |
| table       | 0.4751  | 0.9706  | 0.2678  | 0.6520  |
| sofa        | 0.6932  | 0.9856  | 0.4864  | 0.7352  |
| chair       | 0.8149  | 0.9672  | 0.6757  | 0.8485  |
| toilet      | 0.9139  | 1.0000  | 0.6214  | 0.8000  |
| desk        | 0.2901  | 0.9511  | 0.0759  | 0.4862  |
| dresser     | 0.3544  | 0.9633  | 0.1908  | 0.6147  |
| night_stand | 0.6701  | 0.9608  | 0.5355  | 0.8118  |
| bookshelf   | 0.2548  | 0.8794  | 0.0505  | 0.2872  |
| bathtub     | 0.7533  | 0.9796  | 0.5332  | 0.6735  |
+-------------+---------+---------+---------+---------+
| Overall     | 0.6092  | 0.9646  | 0.4064  | 0.6665  |
+-------------+---------+---------+---------+---------+

+-------------+---------+---------+---------+---------+
| classes     | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 |
+-------------+---------+---------+---------+---------+
| bed         | 0.8681  | 0.9553  | 0.5910  | 0.7146  |
| table       | 0.4706  | 0.9497  | 0.2596  | 0.6427  |
| sofa        | 0.6792  | 0.9665  | 0.4032  | 0.7225  |
| chair       | 0.7962  | 0.9533  | 0.6424  | 0.8232  |
| toilet      | 0.9123  | 0.9931  | 0.6393  | 0.7862  |
| desk        | 0.2962  | 0.9283  | 0.0804  | 0.4718  |
| dresser     | 0.3405  | 0.9083  | 0.1785  | 0.5596  |
| night_stand | 0.6685  | 0.9294  | 0.5227  | 0.7529  |
| bookshelf   | 0.1860  | 0.7730  | 0.0451  | 0.2376  |
| bathtub     | 0.7124  | 0.9592  | 0.3878  | 0.6327  |
+-------------+---------+---------+---------+---------+
| Overall     | 0.5930  | 0.9316  | 0.3750  | 0.6344  |
+-------------+---------+---------+---------+---------+

[Sharpiless]

My experiment enviroment is:

• TITAN RTX
• cuda 11.1
• python 3.8.13
• pytorch 1.9.0+cu111
• mmcv 1.4.0
• mmdet 2.14.0
• mmdet3d 0.15.0
• MinkowskiEngine 0.5.4

[Sharpiless]

It seems like a env error because when I test with privided checkpoint and get the results:

+-------------+---------+---------+---------+---------+                                                                  
| classes     | AP_0.25 | AR_0.25 | AP_0.50 | AR_0.50 |                                                                  
+-------------+---------+---------+---------+---------+                                                                  
| bed         | 0.8528  | 0.9786  | 0.4789  | 0.6000  |
| table       | 0.4681  | 0.9566  | 0.1572  | 0.4736  |                                                                  
| sofa        | 0.6067  | 0.9761  | 0.0737  | 0.3987  |                                                                 
| chair       | 0.7962  | 0.9590  | 0.6565  | 0.8397  |                                                                  
| toilet      | 0.9227  | 1.0000  | 0.4581  | 0.7172  |    
| desk        | 0.2758  | 0.9304  | 0.0258  | 0.3597  |                                                                 
| dresser     | 0.3440  | 0.9450  | 0.0472  | 0.3670  |                                                                 
| night_stand | 0.6761  | 0.9608  | 0.4800  | 0.7686  |                                                                 
| bookshelf   | 0.1038  | 0.7234  | 0.0091  | 0.1241  |                                                                 
| bathtub     | 0.7270  | 0.9592  | 0.1594  | 0.4694  |                                                                  
+-------------+---------+---------+---------+---------+                                                                 
| Overall     | 0.5773  | 0.9389  | 0.2546  | 0.5118  |                                                                  
+-------------+---------+---------+---------+---------

[Sharpiless]

Any suggestions would be greatly appreciated.

[filaPro]

Is everything ok now? It may be caused by updated data preprocessing in mmdet3d. This repo follows their preprocessing before v1.0.

[Sharpiless]

Thanks. I re-preprocessed the dataset and fixed it.