Following the instructions to train a character from scratch does not work well

[zhanchao019]

Hello, I trained a character from scratch according to the readme, but the generated result is not good, especially in the face is blurred and the organs are not clear. Any idea what I did wrong？

[yerfor]

Hi, to make sure whether is something wrong with NeRF, could you refer to checkpoints/<video>/lm3d_nerf/validation_results/validation_*_/frame_*.jpg?

[zhanchao019]

In the path checkpoints/Obama/lm3d_nerf/validation_results/validation_400000_/imgs/frame_7453.jpg

In the path checkpoints/Obama/lm3d_nerf_torso/validation_results/validation_400000_/imgs/frame_7453.jpg

[zhanchao019]

I remember when I was training head in AD-NeRF, there was a Head rendering result for the head. I didn't render the corresponding avatar here。

[yerfor]

Hi, it seems your Head NeRF is not trained well. If you use tensorboard to visualize the training curves, you may find that the loss of NeRF is stuck at a high value (such as 0.05). I have also encountered this problem when reproducing the result of AD-NeRF.

Following this issue, actually, it's a problem of network initialization. They found that in few cases the torso model could not produce proper output due to bad initialization. And in rare cases (with a lower rate than TorsoNeRF) HeadNeRF suffers from bad initialization too.

One solution is to run the command multiple times until good initialization (in my experience, the PNR is greater than 21 in 1000 iterations for a good initialization). Another solution is to use a pretrained Head/Torso model (e.g. torso model trained from another person). Based on my experience, I can get a good initialization with no more than 5 attempts.

[zhanchao019]

OK，let me try

[yerfor]

Yes, in your provided samples, it seems that the head nerf renders nothing, so the torso nerf learns to render the head and torso due to the mse loss with com_imgs. However, the torso nerf cannot model the facial part (it doesn't have 3D landmark as its input), so it tends to render a "mean" face.

[zhanchao019]

I see. So GeneFace is different from the previous method in that the generated head is just a condition fed to torsoNerf, and torsoNerf will generate the whole picture, right? So in this case, the head condition is empty, without previous generated landmark condition, so the torso-NeRF will learn to express body and head together and get the avg face.

[yerfor]

The head image generated by Head NeRF is used to generate the final results. The final_image = head_pred_by_head_nerf + torso_pred_by_torso_nerf, which is equal to AD-NeRF. The difference is that the generated head is also used as the condition of Torso NeRF to prevent head-torso separation artifacts.

During the training of Torso NeRF, when the Head NeRF is trained successfully, the mse loss at the head part is already converged, so only the torso part is learned by the Torso NeRF. In your case, as the Head NeRF renders nothing, the Torso NeRF also needs to predict the head part. However, since the input space of Torso NeRF doesn't have 3D landmark, it cannot model the variance in facial motion. As a result, the head part predicted by the Torso NeRF is blurry.

In summary, my suggestion is to run the train_head_nerf commandline several times until good initialization (or initialize with a pre-trained head nerf of another target person), then run the train_torso_nerf commandline several times until good initialization (or initialize with a pre-trained torso nerf of another target person).

[zhanchao019]

Actually, when I initialize with a pre-trained head nerf of another target person to train_torso_nerf, the init result is very weird while psnr is 25.

This was annoying because I started the experiment over and over again and the initialization didn't work. Is there any solution? QAQ

[yerfor]

Hi, we use vanilla NeRF in the current GeneFace version, whose performance is widely proven heavily rely on a good initialization. Exchanging the NeRF-based renderer with advanced NeRF techniques may help. Otherwise, we have to try several times until a good intialization is rolled.

I'd like to make a forecast: our recent work named GeneFace-S, which is an upgrade version of GeneFace, could fully handle this bad initialization problem by using instant-ngp as the backend of NeRF-based renderer. Besides, it could provide more accurate and stable lip shape to various voices; it could be trained 4x faster (only 10 hours for a whole model) and could infer in real time (25fps) on RTX2080! Please stay tuned!