Continuing training on your shared model

[Vijayue]

Thank you for your shared model! And now I'm continuing training on the shared model using the voxceleb2 sub-datasets (part_b part_c and part_d, about 380k videos, the paper said using 280k videos).
After every epoch, I evaluated the model but seems that the performance is gradually worse.
Although the training losses are decreasing, the PSNR of generated videos is decreasing, and the visual quality is also worse. It's so strange.

Do you have any thinking about it? Could you share more training details you thought necessary? Thank you a lot.

up: shared model
below: the continuing training model
You can see the background is moving using my model.

[zhengkw18]

Maybe you didn't get the right cropping so the scene is moving? I'm not sure. Or the dataset transfer requires more training. I didn't try VoxCeleb2, but my continue training on VoxCeleb1 is ok. You can check the reconstruction samples after each epoch to see whether such problem exists, or whether there are ridiculous keypoints residing far from the face.

Actually I didn't use training tricks. Maybe the only thing needed is to increase the learning rate if you have large batch.

[Vijayue]

Thank u a lot! I have solved the problem, it's the problem of the datasets.

And now I have another question about the training process. Now I'm training on vox2 from the scratch, has been two days about 10 epochs, but the vis results demonstrate there is no difference between source and prediction.

Considering the cost of training(time and money), I want to know whether my training params is wrong, or training time is not enough. Could you share with me, you can see some difference between source and predictions after how many epochs?

Thank you.

[zhengkw18]

When the headpose loss (H in log) decreases from 200+ to 50, the prediction pose starts to align with driving. On Vox1 10 epochs is enough (for num_repeates=100), but Vox2 may take more. At first the reconstruction demonstrates much distortion, as the training goes on, the distortion is weaker and the expressions start to align with driving, too.

[Liangtian96]

您好！想问一下设置num_repeates=100和将num_repeates设置成1然后epoch设置*100倍有区别吗？不是很清楚为什么一个epoch下还要设置多个num_repeates

[zhengkw18]

没有区别。因为默认开启id sampling，会对identity均匀采样，vox1共1152个id，也就是一个repeat只有1152组样本，通过增加repeat来减少epoch数。