-
Notifications
You must be signed in to change notification settings - Fork 516
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Model producing blank images with dataset of 128x128 or larger images #98
Comments
Hello, I'm having a very similar problem to @dpmcgonigle , even when using the exact same setup as the original paper. Would be amazing to get some clarification as to the possible issues behind this problem. Btw, @dpmcgonigle have you been able to resolve this in the meanwhile? |
@guilherme-pombo , this is something that we have not yet resolved. I should have noted that both the training loss and test loss continue to decrease even when sampling blank images during training, interestingly enough. I have noticed that with some of our experiments, there are all-blank sets of sample images for a number of "epochs" before actually "breaking out" and producing good images. So, I'm wondering if it is just a matter of time before "good images" are produced. One piece of evidence against this hypothesis I think is that the trained model producing all blank images is not invertible from Z -> X -> Z with a successive model.decode() then model.encode(), though it is invertible in the other direction X -> Z -> X with model.encode() on a real image, followed by a model.decode(). I am very curious if anyone else may have insight into this problem. |
@guilherme-pombo, just wanted to let you know I've had limited success using:
|
@@@@@@@ |
Congratulations! By the way, is the reversibility of your glow model good? |
If the exact inputs are desired to be constructed, the output z of different scales should be gathered. |
Code for pre-training, and the pre-trained models on the image size of 256, are now available at https://github.com/Hzzone/AgeFlow. |
Hello,
I have tried training several 256x256 datasets with the Glow model using hyperparameter configurations as similar to the CelebA configuration that your team used, and all I'm getting are blank images when trying to generate samples with any temperatures(standard deviation).
For instance, I have LSUN-Tower 256x256 training right now (~708K images), on epoch 800 (as epoch is defined by this code base; approaching an entire pass through the entire dataset). For this experiment I am using 6 Levels, with 32 Steps per level, learning rate 0.001 with local_batch_train of 2 images per GPU (using 4 GPUs), which is derived from n_batch 128 using the default anchor of 32, and I am using the affine coupling layer. There is no y-conditioning for the experiments I'm running (all of the same category), and I’m using n_bits_x of 8.
I have run several other experiments similar to this with other datasets, and additive coupling layer, for longer times, and all have produced blank image samples. It appears any dataset larger than 64x64 is giving me trouble. Is this something that your team has run into at all with larger images? I'm wondering if it will eventually "break out" of this issue if I let it run long enough, or if I need to tweak the hyperparameters.
Thank you very much for your time.
Sincerely,
Dan McGonigle
The text was updated successfully, but these errors were encountered: