Training time

[rohitrango]

Hi,

Thank you for the amazing work!

I was wondering how long does the synthetic data training take, considering there is no DataParallel/DistributedDataParallel implementation.
Thank you in advance!

[rohitrango]

Just in case, do you also have training loss curves, etc?

[akashsengupta1997]

Hi - sorry for the late reply!

Synthetic training takes around 6 days on a 2080Ti. Unfortunately, it is quite slow due to the matrix-Fisher distributions used - since we need to do rejection sampling from these distributions. (Future work could look at using better distributions over SO(3) with faster sampling.)

I do have training logs saved - I will email them to you by the end of the week (please remind me if I haven't done it)

[rohitrango]

Thank you for your response.

Can you email them to rohitrango@gmail.com?

[rohitrango]

Another small question.

The training parameters that is mentioned in the paper mentions 150 epochs in total. But the default config file mentions more iterations (300) with different number of iterations for loss stages 1 and 2 (different from that in the paper). Which configuration works best? TIA.

[akashsengupta1997]

Hi,

The epochs in the default config is set higher than needed - I usually track the train/val curves and early-stop the experiment when it seems like they have converged. IIRC the metrics in the paper were achieved with ~150 epochs. The weights that are released with the repo were trained for the full 300 epochs, mostly because I started the experiment and forgot about it for a while 😄 Training for 300 epochs probably will not perform much better on real test data than 150 epochs.