How to shut it up?

[bigboss97]

I used the colab and it worked beautifully, THANKS!
I have a problem. Whenever the audio has a silence piece the original mouth movement remains unchanged.
I've tried with "mmm" to "shutup". But it had no effect. So I used "ha ha ha" to minimize the lip movement.
Is there a way to close the mouth for silence?

[prajwalkr]

Did you try once with the non-GAN model? It performs better on silences

[bigboss97]

Did you try once with the non-GAN model? It performs better on silences

I'm not familiar with that. How can I do that in colab?

[prajwalkr]

Just download the corresponding checkpoint and pass it as an argument in the same inference command.

[HusainKapadia]

Just a suggestion...perhaps to improve performance on silence you can add some simulated data to the training data set, where:

1. The audio inputs would be silence or add some low DB noise to it (to avoid Mel bands being NaN)
2. The corresponding video could be created with an image of any random person with a neutral expression (of course the length's of the audio and video should match)
3. Train the model with this extra data along with the LSR2 dataset specified in the README. Or you could fine tune the model with the pre-trained weights.
4. Run inference with new model.

PS: this is not something I have tried yet, but if this is a limitation of the existing model, this could be one way of improving it, I believe
Would you agree @prajwalkr ?

[prajwalkr]

Maybe it can help. Maybe using some short range temporal info in the generator might help even better. We are yet to experiment more on this.

[prajwalkr]

Re-open if needed.