100 epochs with 10,000 images from celebA... still noise? #4
Comments
|
Obviously - something is wrong. I trained on celebA (didn't put up the weights, as it isn't really 'art') and got pretty good results (recognizable as a face) pretty early on... Make sure that your display hasn't crashed and isn't updating - run EDIT: sorry - didn't mean to close this issue :P |
|
Okay nvm... I removed two folders in my "art-DCGAN-master" directory that I believe may have been screwing with the process... one named "images" and another named "folder"... both empty. |
|
Hmmm... I wonder if it's still supposed to look like this after 100 epochs: https://imgur.com/a/VOG63 Here is a portion from the logs: I think maybe I'm missing something? |
|
Also... what happens if I run the command "DATA_ROOT=landscape dataset=folder ndf=50 ngf=150 th main.lua" AFTER I have already run it (choosing not to load from a checkpoint)? Does it just start the process over completely or does the file in the "cache" folder get involved? |
|
delete the contents your cache folder - it builds the dataset into arrays to be used by the network, i think you are only training on a small portion of the dataset (the 13 number in the logs should be a lot larger - it's just your batches). Delete the contents of |
|
How big should it be? I deleted the file and now it's at 19... |
|
What's your batch size and number of images in your folder? |
|
Maybe my dataset for the landscape images is too low? There are only 1262 or so images and the batch size is just as default - 64 |
|
Oh yes... just for clarification: |
|
But I didn't install luarocks cudnn or cunn stuff cuz that made it run into errors when I ran the training code. |
|
I think I'm gonna try with all 200,000+ images from celebA and see what happens... but I'm pretty sure I'm not getting the results as they are meant to be. |
|
Hmm.... I doubt the cudnn has anything to do with it; although you may run into some errors loading from saved models (don't quote me on that)... I think that the dataset size is the problem; I've gotten some very strange results when trying to train on data under ~3,000 images... Let me know what you get with celebA - as that should definitely work. If it doesn't, send me your entire project folder (minus the dataset, maybe) on google drive or something, and I'll take a look myself. Keep in mind that the project is currently undergoing a total overhaul in Keras, and is being reimplemented with a better model and in python/keras instead of torch, so if we're unable to solve your problems now, they shouldn't be an issue anymore in a week or two after the update. |
|
Okay, great to hear :). I will report back shortly. |
|
Okay it works great now!.. although, I had to adjust the layers from 50:150 to 20:120 - otherwise, my discriminator overpowers my generator every single time and it just remains in noise forever. |
|
Ah - nice, glad you got it working. Also; yeah, that makes sense. GANs are really "hard to train" - meaning that if you don't set all the hyperparameters just right, it'll screw everything up. That's actually the reason I wanted to see your logs, usually if the discriminator wins over the generator, d_loss goes down and hangs around 0.00001 or a similar low value... The discriminator does have the easier job, so it often wins over unless you severely handicap it's number of layers. You might have to play around with different numbers of filters per network when moving to different datasets, because the 20:120 ratio might not work for all of them. |
Hi, thanks very much for adding a more layers so that the networks would be able to generate higher res images...
I'm a bit confused about how to go about training properly. I put 10,000 images from "img_align_celebA" into the landscape/images folder and ran "DATA_ROOT=landscape dataset=folder ndf=30 ngf=90 th main.lua", but I'm still getting almost pure noise in the localhost:8000 display... is this normal?
The text was updated successfully, but these errors were encountered: