Abnormal chamfer metric reproduced on ShapeNet

[chengzhag]

Hi Kyle:

I'm trying to reproduce the results with original settings (model type: ldif, batch size: 24) on ShapeNet. Watertight mesh generation was done using code from OccNet before preprocessing with your code make_dataset.py. The data is randomly split into train, val, test set like in the paper. The sizes of the meshes are unchanged, which are whin a cube with side length 2.

However, after 200k iterations of training (which is less than 1M in the paper), I've got the following results on the val split:
mean: IoU=81.78, Chamfer=0.03, F-Score=90.25
Comparing to the results in your paper (mean: IoU=90.00, Chamfer=0.4, F-Score=92.20) and other works, the IoU and F-Score metrics seem Alright considering less training. However, Chamfer metric is abnormally low.

I can't find any mistake before data preprocessing. It would be kind if you can give some hints about the process. It would be better if you can publish pretrained model, watertight mesh generation code and preprocessed data.

Best wishes
Cheng

[chengzhag]

By the way, I'm using the following command to evaluate on val split:
python eval.py --dataset_directory ldif/data/ShapeNet/ldif_dataset/ --experiment_name {example} --split val --use_inference_kernel --result_directory {example} --save_ldifs --save_results --save_meshes --eval_frac 0.1
It is worth mentioning that param 'eval_frac' is used. Don't know if this has anything to do with the behavior.

A ShapeNet model and its processed watertight mesh look like below:

[chengzhag]

Update: I tried a full test on val split. The result is the same: IoU=81.81, Chamfer=0.045, F-Score=90.38.

[kylegenova]

Hi Cheng, thanks very much for bringing this to our attention.

For display purposes the chamfer distance of all methods is scaled up (equation in the appendix). The code currently scales it up by a factor of 100, but looking at the output logs for the paper, it looks like we really used a scale of 1000; this is a mistake. Any chamfer you have previously computed can be multiplied by 10x. I will push a patch that increases the scale from 100x to 1000x in ldif/inference/metrics.py.

[kylegenova]

Commit 3482786 should fix this.

As an aside, the F-Score and IoU at that stage seem fairly expected to me. The IoU and F-Score should continue to climb slowly until training completes. To be honest, training never really 'completes'- even the LDIF models used for the paper were continuing to improve (albeit very slowly) on the val set, we just stopped training at some point. I have not seen an LDIF on ShapeNet fully converge. SIF converges much more quickly though and val performance flattens out.

[chengzhag]

Thanks for your quick fix and the kind reminder about the training process!

However, the training speed is not so ideal on a 1080Ti GPU. The 200k iterations' training took me more than 4 days to complete. It looks like that the bottleneck is on the reading speed of my hard disk which I can't improve easily.
May I ask what kind of hardware did you train on and how long have you taken to complete the 1M iterations' training?
Furthermore, may I ask when or if will the checkpoint be released?

[kylegenova]

Hmm, that is several times slower than the performance on the hardware I tested on (a V100 with the data on an SSD). The IO code was newly written for the open source release. For the paper we trained on a cluster and had a more optimized IO pipeline, where more of the processing was done before training rather than on-the-fly.

Given that the input process code is simpler than the previous pipeline and does a nontrivial amount of work, it is not too surprising it can bottleneck on some hardware. I think I can resolve this issue, but it will take some time.

In order to help make sure your issue is addressed, can you tell me whether the code appears to be bottlenecked on IO throughput (the drive is bottlenecked filling read requests), IO latency (there aren't enough concurrent read requests to keep either the drive or CPU busy), or CPU performance (the CPU is at 100%)? My guess is that it is IO throughput, but since it is not occurring on my machine I can't be sure.

I can share a more fully trained checkpoint with you directly, if you would like, but the original checkpoints that were used to write the paper are incompatible with this code base (they have dependencies that could not be open-sourced). I think the long-term fix to this problem is probably multi-gpu support + a more optimized IO pipeline (like tf-records coupled with less data read per example)

I also updated #2 regarding the checkpoint release.

[chengzhag]

CPU usage is around 20% on a 16 core CPU i7-9800X. GPU is not fully running with usage like below:

The data is stored on a central server with a GbE network shared by several GPU servers. So it is hard to know the exact IO latency. IO throughput is around 80MB/s seen from the network usage, which should be the bottleneck. Your guess should be quite accurate.

I'll try some improvement on the storage bottleneck. At the same time, a fully trained checkpoint would be great. How can I get in touch with you personally? Thank you for your patience by the way!

[chengzhag]

Update: I'm trying to get in touch with you personally though your princeton.edu email. Thanks in advance!