New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

How many epochs will be take when training celeba? #14

Closed
weitaoatvison opened this Issue Jul 17, 2018 · 9 comments

Comments

Projects
None yet
5 participants
@weitaoatvison
Copy link

weitaoatvison commented Jul 17, 2018

Hi, I'm trying to train a model on celeba datasets with your code. And I trained the model with the command line below:
mpiexec -n 8 python train.py --problem celeba --image_size 256 --n_level 6 --depth 32 --flow_permutation 2 --flow_coupl ing 0 --seed 0 --learntop --lr 0.001 --n_bits_x 5

I just modify the gpu numbers to 8 because I only have a 8 cards server. Now, I have trained for 4days with 1300epochs, however, the results seems not good when I check the example images generate by code. So, could you tell me how many epochs it takes when you trained on celeba dataset? @prafullasd

@voa18105

This comment has been minimized.

Copy link

voa18105 commented Jul 17, 2018

Same for me, it does not converge to results presented in the article. How long it should be trained?

@pclucas14

This comment has been minimized.

Copy link

pclucas14 commented Jul 18, 2018

@weitaoatvison @voa18105 could you post pictures of the samples you get ?

@voa18105

This comment has been minimized.

Copy link

voa18105 commented Jul 19, 2018

Sure, here my beauties :-) screenshot_20180713-134952

@weitaoatvison

This comment has been minimized.

Copy link

weitaoatvison commented Jul 19, 2018

@voa18105 My result is similar to yours. 1300epoch
epoch_1300_sample_5

@voa18105

This comment has been minimized.

Copy link

voa18105 commented Jul 19, 2018

Actually, your's is better. So, it improves.

@dpkingma

This comment has been minimized.

Copy link
Contributor

dpkingma commented Jul 20, 2018

@voa18105 @weitaoatvison
The samples we show in the paper are after about 4000 training epochs. As shown in figure 8 of the paper the (subjectively) most realistic-looking samples were obtained with lower temperature. The samples from figure 1 are with a temperature of 0.7. The training script creates samples with various temperatures; for teperature 0.7 make sure to look at the right one, namely the files "*_sample_4.png". See https://github.com/openai/glow/blob/master/train.py#L47

@dpkingma

This comment has been minimized.

Copy link
Contributor

dpkingma commented Jul 20, 2018

@voa18105 @weitaoatvison

P.S. I encourage you to look at temperature=0.7 samples at 1300 epochs. They should look better than the one shown above, which was probably with temperature=1.0.

Cheers

@weitaoatvison

This comment has been minimized.

Copy link

weitaoatvison commented Jul 23, 2018

@dpkingma Thanks. I will try!

@winwinJJiang

This comment has been minimized.

Copy link

winwinJJiang commented Jul 26, 2018

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment