-
Notifications
You must be signed in to change notification settings - Fork 34
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
How to continue training without replacing the previous training? #19
Comments
Sorry, I didn't get it at all, what do you mean by "overwriting images"? And why do you want to train (or fine-tune?) two times. If you have two datasets, just fine-tune to them independently using a single meta-learned checkpoint |
I want to train two sets of the same person separately because if I load the whole dataset, dataset are images-cropped generated by preprocess data.py images are expressions or the face of a person. driver is to take a video driver, a chekpoint and produce a video with driver.py When selecting images I mean which images from the data set are chosen to make the output video with driver.py By overwriting images I mean overwriting expressions or faces. If data set A has different illumination to data set B, if I refine a meta-learned model with data set A with python3 train.py, then I repeat but with data set B, when making a driver only images are seen with the illumination of B, then B overwritten A. I don't want to train, I want to tune. |
This means that you're doing something wrong: GPU memory doesn't depend on the dataset size. Just use smaller batches. For example, with a batch size of 1 you can fine-tune on as many images as you want. As I understood, you're trying to fine-tune a meta-learned checkpoint to dataset A, then take that fine-tuned model and fine-tune it further to dataset B. Well, we never tried that. I don't know if that will even work -- that's a research question. You'll probably need to modify the code for that, and it's entirely at your risk, I'm afraid I can't help here. |
ok I understand, another thing, can I make the checkpoint smaller?, output is always 1 gb size. |
Yes, it's not hard (just don't include discriminator, embedder, optimizer state etc. in the checkpoint) but for that you'll have to modify the code yourself. |
How to continue training without replacing the previous training?
I have images of a face divided into part1 and part2
first I train part1 of images of face.
when I finish training and do a driver, you can see the face of the images in part1
Then I want to add the images from part 2.
then I load the pth and train from there with the images of part 2
The problem is that when I make a driver, only the images of part2 are chosen, it is as if part2 were to overwrite part 1.
What I expected was to have more variety of expressions, so expressions from part 1 and part 2 are chosen.
How to avoid overwriting images or expressions from a previous training session?
The text was updated successfully, but these errors were encountered: