stylegan2 produces color splats

[abesmon]

I'm trying to run stylegan2 configuration, but it produces almost random color splats. What can it be?
Same results appeared on google collar pro (p100)
and on paper space gradient (nvcr.io/nvidia/pytorch:21.10-py3 docker + quadro m4000)

[jlmarrugom]

Which parameters did you use? What is the resolution of your dataset? How many Kimg have passed to produce those results?

[abesmon]

default parameters from Colab:
10 kimg per tick
learning rate 0.002
batch 64 img

I tried different resolutions from 128 (examples above) to 512. All making same results
Also I tried different datasets once tried with simply 10 images (examples above) and tried with 35k imgs. Each making these color splats :/

Examples above are 30k images and 60k images passed

But I got same results even after 140k images past. Splats are just in different places

[abesmon]

here's 120k

and 148k

[abesmon]

above are 128^2 and with 10 images dataset

[xl-sr]

Depending on the dataset, StyleGAN2 can sometimes (but rarely) still become unstable. What dataset are you using here?

[abesmon]

@xl-sr I'm using my own Pokemon dataset based on original pokemons + ~3500 fanart pokemons from deviantart + augmentations of all above (hue+mirror) so in the end its like 35k images of pokemons

[abesmon]

Main question is: if I'll continue to teach this network. Will it produce more sane results in future or is everything above looks like sort of bug?

[xl-sr]

Nope, continued training will not result in anything useful, the optimization already spiraled out of control.
The oversaturation indicates that your output magnitude is very large, likely due to gradient peaks at the beginning of training.

You can try different configurations for the SG backbone, e.g G.synthesis_kwargs.architecture = 'orig' or 'resnet' (default is 'skip'). Another thing to try is lowering the learning rates. Lastly, you can still resort to the FastGAN backbone which should work fine in your case.

[abesmon]

@xl-sr yeah, FastGAN works just fine! I was just curious about difference in these configs outputs :)

I'll try different architectures and write back here about any differences. Thank you for advice :)

[xjdeng]

I'm getting the same color splats with every dataset I'm trying on stylegan2 but fastgan seems to be working fine.

I guess I'll stick with stylegan2-ada for the ones I want trained using stylegan and only use fastgan with this implementation.

[xl-sr]

Training projected StyleGAN2 on large datasets eg. LSUN will result in better and faster performance than StyleGAN2-ADA. On small datasets (which I suspect you tried), I did not do much testing, so you might need to do some tuning yourself eg. adapt the learning rate, add R1 regularization, etc.

[manzke]

Just wanted to say I can reproduce it. Haven't got stylegan2 to run. My dataset works with all other implementations, but not PG.

Training options:
{
"G_kwargs": {
"class_name": "pg_modules.networks_stylegan2.Generator",
"z_dim": 64,
"w_dim": 128,
"mapping_kwargs": {
"num_layers": 2
},
"channel_base": 32768,
"channel_max": 512,
"fused_modconv_default": "inference_only"
},
"G_opt_kwargs": {
"class_name": "torch.optim.Adam",
"betas": [
0,
0.99
],
"eps": 1e-08,
"lr": 0.002
},
"D_opt_kwargs": {
"class_name": "torch.optim.Adam",
"betas": [
0,
0.99
],
"eps": 1e-08,
"lr": 0.002
},
"data_loader_kwargs": {
"pin_memory": true,
"prefetch_factor": 2,
"num_workers": 3
},
"training_set_kwargs": {
"class_name": "training.dataset.ImageFolderDataset",
"path": "/content/drive/MyDrive/projected_gan/dataset-1024x1024.zip",
"use_labels": false,
"max_size": 1267,
"xflip": false,
"resolution": 1024,
"random_seed": 168369
},
"num_gpus": 1,
"batch_size": 64,
"batch_gpu": 4,
"metrics": [],
"total_kimg": 25000,
"kimg_per_tick": 4,
"image_snapshot_ticks": 5,
"network_snapshot_ticks": 5,
"random_seed": 168369,
"ema_kimg": 20.0,
"resume_pkl": "/content/drive/MyDrive/projected_gan/output/00001-stylegan2-dataset-1024x1024-gpus1-batch64/network-snapshot.pkl",
"ema_rampup": null,
"restart_every": 9999999,
"loss_kwargs": {
"class_name": "training.loss.ProjectedGANLoss"
},
"D_kwargs": {
"class_name": "pg_modules.discriminator.ProjectedDiscriminator",
"diffaug": true,
"interp224": false,
"backbone_kwargs": {
"cout": 64,
"expand": true,
"proj_type": 2,
"num_discs": 4,
"separable": true,
"cond": false
}
},
"run_dir": "/content/drive/MyDrive/projected_gan/output/00001-stylegan2-dataset-1024x1024-gpus1-batch64"
}

Output directory: /content/drive/MyDrive/projected_gan/output/00001-stylegan2-dataset-1024x1024-gpus1-batch64
Number of GPUs: 1
Batch size: 64 images
Training duration: 25000 kimg
Dataset path: /content/drive/MyDrive/projected_gan/dataset-1024x1024.zip
Dataset size: 1267 images
Dataset resolution: 1024
Dataset labels: False
Dataset x-flips: False

Creating output directory...
Launching processes...
Loading training set...

Num images: 1267
Image shape: [3, 1024, 1024]
Label shape: [0]

[joel-simon]

I had this issue too for a diverse dataset on 512px stylegan2 and setting the glr and dlr to .0007 fixed it for me.