Can you help me to create a budget to train a model?

[davidmartinrius]

Hello,

First of all, thanks for your contribution and sharing this very valuable code. You have done a fantastic job.

I have two questions:

1- How long did it take to train a model with 10 x A6000 GPUs (assuming each one was 48GB)?
2- How much did it cost to rent the GPU service from the time you started until the model training was finished?

I understand that you can't share a pre-trained model. For you that is a lot of hours of dedication and money.

Although it would help me to know those two points to make a rough budget and train my own model.

I know I could make estimated calculations, although the reality is always different in practice.

Thank you!

David

[ghost]

1 - This also depends on your dataset, how much your dataset is? for LRW that could be more than a month to train sycnnet & wav2lip, the hardest part is syncnet which could make more than 3 weeks for tuning parameters
2 - you can calculate price for two month in at least.

[davidmartinrius]

Sorry, I overlooked it. I took the AVSpeech dataset for granted.
Although if you say that with LRW it can be more than a month and that I can calculate a price for a couple of months I have an idea of the cost.

Could you suggest me some good and ""economical"" service that offers uninterrupted and reliable service?

The GPU's from google, amazon, and bigger companies are quite expensive.

Thanks!

[ghost]

you should request spot instance of azure or preemptive of google, which can reduce 90% cost of your machine

[davidmartinrius]

Great. I didn't know about this kind of service. I can't wait to try it :D

I will have to do a lot of checkpoints during the trainings because I have seen that the service can be interrupted depending on demand. It totally pays off though because of the cost savings.

I am very grateful for your help.
Thank you very much! 😊

[NikitaKononov]

Great. I didn't know about this kind of service. I can't wait to try it :D

I will have to do a lot of checkpoints during the trainings because I have seen that the service can be interrupted depending on demand. It totally pays off though because of the cost savings.

I am very grateful for your help. Thank you very much! 😊

Unfortunately, you won't be able to get success in training syncnet without modifications to this repo

[ghost]

Great. I didn't know about this kind of service. I can't wait to try it :D
I will have to do a lot of checkpoints during the trainings because I have seen that the service can be interrupted depending on demand. It totally pays off though because of the cost savings.
I am very grateful for your help. Thank you very much! 😊

Unfortunately, you won't be able to get success in training syncnet without modifications to this repo

Absolutely

[davidmartinrius]

Great. I didn't know about this kind of service. I can't wait to try it :D
I will have to do a lot of checkpoints during the trainings because I have seen that the service can be interrupted depending on demand. It totally pays off though because of the cost savings.
I am very grateful for your help. Thank you very much! blush

Unfortunately, you won't be able to get success in training syncnet without modifications to this repo

@NikitaKononov Thanks for letting me know this. Some months ago I asked it in
#16

@hannarud told she used BCEwithLogits to train color_syncnet_train.py
Unfortunately, I got the same result and I have not enough experience to solve it.

@NikitaKononov Can you help me to make the appropriate modifications to train it?

Thank you!

[ThetaRgo]

Great. I didn't know about this kind of service. I can't wait to try it :D
I will have to do a lot of checkpoints during the trainings because I have seen that the service can be interrupted depending on demand. It totally pays off though because of the cost savings.
I am very grateful for your help. Thank you very much! 😊

Unfortunately, you won't be able to get success in training syncnet without modifications to this repo

but how? are we disscuss at github? or a private group chat ?

[NikitaKononov]

but how? are we disscuss at github? or a private group chat ?

Author had released stripped down version of his developments for some reason. Maybe NDA

[ThetaRgo]

but how? are we disscuss at github? or a private group chat ?

Author had released stripped down version of his developments for some reason. Maybe NDA

yes, this project already cost me some weeks,the loss do not drop around 0.51 i think the dataset process is the biggest reason,read issue ask for help . some bugs have find and can be bugfixed but not. so this project is just make a contribute to power stations 。

[davidmartinrius]

Hi everyone,

I think this project is not useful. There are other functional, better and scalable projects. You can search for better lip sync alternatives in github and you will find these.

The best projects that I found are:

DINet (audio input). This project was released a month ago. Unfortunately the autor said that has not time to keep it updated, but there is a functional version that you can use and train.
I am about to train a model. I downloaded the AVSpeech dataset, I used OpenFace to detect the landmarks and this week I will start a training with a little part of the dataset to test the training.
Anyway, there is a pretrained model that you can use if you want a demo.
https://github.com/MRzzm/DINet

Geneface (NeRF). the author is very attentive to new incidences and will answer you. It is a project that for the moment is kept up to date:
https://github.com/yerfor/GeneFace

Wav2lip with GFPGAN:
It is the original project with face restoration. The inference is quite slow because it has to restore all the frames. But the quality is good enough at all.
https://github.com/ajay-sainy/Wav2Lip-GFPGAN

Also there is a list of new lip sync projects at:
https://github.com/YunjinPark/awesome_talking_face_generation

I hope this is helpful for you 👍

David Martin Rius

[ThetaRgo]

Hi everyone,

I think this project is not useful. There are other functional, better and scalable projects. You can search for better lip sync alternatives in github and you will find these.

The best projects that I found are:

DINet (audio input). This project was released a month ago. Unfortunately the autor said that has not time to keep it updated, but there is a functional version that you can use and train. I am about to train a model. I downloaded the AVSpeech dataset, I used OpenFace to detect the landmarks and this week I will start a training with a little part of the dataset to test the training. Anyway, there is a pretrained model that you can use if you want a demo. https://github.com/MRzzm/DINet

Geneface (NeRF). the author is very attentive to new incidences and will answer you. It is a project that for the moment is kept up to date: https://github.com/yerfor/GeneFace

Also there is a list of new lip sync projects at: https://github.com/YunjinPark/awesome_talking_face_generation

I hope this is helpful for you 👍

David Martin Rius

thank you! David,you are a real githuber

[davidmartinrius]

By the way, I forgot to add another interesting project that you can use. Check the above message, I edited it.

[ThetaRgo]

got it!