Any data on expected training time? Also, Are there plans for realtime visualized output?

[ExponentialML]

Hello. First, I would like to thank you for such great, fully fleshed out project.

In regards to training, is there any data on how long it should generally take per GPU? If we're talking about a GPU with 8GB of memory, how long would it take to finish training on a small to medium sized data set?

This one kind of ties into the training time, but are there any plans for realtime visualization of training data? If the training does take long periods of time, it would be beneficial to see the results updated at a faster rate.

Any input is appreciated. Thanks!

[AliaksandrSiarohin]

Hi, thank you for your interest,
I usually train on 2 Titan X with 12GB of memory. It take 2 days for dataset like TaiChi and VoxCeleb. For simple dataset like Nemo, training can be done in 24 hours. With 1 gpu number should be multiplied by 2.

The problem with visualization is that it take a lot of time to compute. If you need more frequent visualizations you can change the parameter num_repeats

first-order-model/config/taichi-256.yaml

Line 93 in 720bdd8

num_repeats: 150

. Basically the number of passes over all the videos is num_epochs * num_repeats. You can decrease number of repeats and increase number of epochs. Because visualization is drawn after each epoch, you will get more frequent visualizations at a cost of some efficiency.

Regards, AS

[ExponentialML]

Hi, thank you for your interest,
I usually train on 2 Titan X with 12GB of memory. It take 2 days for dataset like TaiChi and VoxCeleb. For simple dataset like Nemo, training can be done in 24 hours. With 1 gpu number should be multiplied by 2.

The problem with visualization is that it take a lot of time to compute. If you need more frequent visualizations you can change the parameter num_repeats

first-order-model/config/taichi-256.yaml

Line 93 in 720bdd8

num_repeats: 150

. Basically the number of passes over all the videos is num_epochs * num_repeats. You can decrease number of repeats and increase number of epochs. Because visualization is drawn after each epoch, you will get more frequent visualizations at a cost of some efficiency.

Regards, AS

Excellent. You rock!

[kartikJ-9]

Can i train this on colab. How many videos I should keep for train and test? I want the output quality to be 256/512 resolution. I can see that the fashion dataset had 500 videos for train and 100 for test. My dataset is also similar to fashion with a person in the frame. So will I be able to train the model in colab if I use 500,100 for train,test? Please help me with this. Thanks

[AliaksandrSiarohin]

Most likely not possible, colab gpus is rather slow. And it will shut down and kick you out after 24 hours.

[MentalGear]

@kartikJ-9 Have you made progress with training a higher res model on colab or elsewhere? Would appreciate if you could share what you tried.