Get Strange results when training a X3 upsacle ESRGAN model

[JunbinWang]

Dear all,

When I try to train a ArtefactCancelling ESRGAN model, I get some result images with strange patterns.

I put some sample images here:

I first train the model with only mae loss as the script below.

lr_train_patch_size = 50
layers_to_extract = [5, 9]

scale = 3
hr_train_patch_size = lr_train_patch_size * scale

rrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':32, 'G0':32, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)

from ISR.train import Trainer
loss_weights = {
  'generator': 1,
  'feature_extractor': 0.0,
  'discriminator': 0.0
}
losses = {
  'generator': 'mae',
  'feature_extractor': 'mse',
  'discriminator': 'binary_crossentropy'
}

log_dirs = {'logs': './logs', 'weights': './weights'}

learning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}

flatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}

trainer = Trainer(
    generator=rrdn,
    discriminator=None,
    feature_extractor=None,
    lr_train_dir='train_LR',
    hr_train_dir= 'train_HR',
    lr_valid_dir='valid_LR',
    hr_valid_dir='valid_HR',
    loss_weights=loss_weights,
    learning_rate=learning_rate,
    flatness=flatness,
    dataname='hotel',
    log_dirs=log_dirs,
    weights_generator=None,
    weights_discriminator=None,
    n_validation=40,
)

trainer.train(
    epochs=100,
    steps_per_epoch=700,
    batch_size=16,
    monitored_metrics={'val_generator_PSNR_Y': 'max','val_generator_loss': 'min','val_loss': 'min'}
)

Then I train the model with the following configuration ：

lr_train_patch_size = 50
layers_to_extract = [5, 9]

scale = 3
hr_train_patch_size = lr_train_patch_size * scale

rrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':32, 'G0':32, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)
f_ext = Cut_VGG19(patch_size=hr_train_patch_size, layers_to_extract=layers_to_extract)
discr = Discriminator(patch_size=hr_train_patch_size, kernel_size=3)

from ISR.train import Trainer
loss_weights = {
  'generator': 0.1,
  'feature_extractor': 0.8,
  'discriminator': 0.1
}
losses = {
  'generator': 'mae',
  'feature_extractor': 'mse',
  'discriminator': 'binary_crossentropy'
}

log_dirs = {'logs': './logs', 'weights': './weights'}

learning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}

flatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}

trainer = Trainer(
    generator=rrdn,
    discriminator=discr,
    feature_extractor=f_ext,
    lr_train_dir='train_LR',
    hr_train_dir= 'train_HR',
    lr_valid_dir='valid_LR',
    hr_valid_dir='valid_HR',
    loss_weights=loss_weights,
    learning_rate=learning_rate,
    flatness=flatness,
    dataname='hotel',
    log_dirs=log_dirs,
    weights_generator='weights/rrdn-C4-D3-G32-G032-T10-x3/rrdn-C4-D3-G32-G032-T10-x3_perceptual_epoch099.hdf5',
    weights_discriminator=None,
    n_validation=40,
)

trainer.train(
    epochs=400,
    steps_per_epoch=500,
    batch_size=16,
    monitored_metrics={'val_generator_loss': 'min', 'val_generator_PSNR_Y': 'max','val_loss': 'min'}
)

Can somebody give me some hints about what may cause the unpleased image patterns?

Best Regards,
Even

[slee5777]

@JunbinWang checkout https://distill.pub/2016/deconv-checkerboard/
It is an interactive website so you can adjust stride and size to experiment with different combinations.

[JunbinWang]

@JunbinWang checkout https://distill.pub/2016/deconv-checkerboard/
It is an interactive website so you can adjust stride and size to experiment with different combinations.

Hi @slee5777,

Thanks for your reply!

So you suggest that this is caused by wrong Upsampling factors? Maybe I should not upscale the LR image to x3 ?

[cfrancesco]

@JunbinWang checkout https://distill.pub/2016/deconv-checkerboard/
It is an interactive website so you can adjust stride and size to experiment with different combinations.

This is pretty cool. I did struggle to get rid of those artefacts.
I'm not sure though if this entirely applies to the models used here, as they use a pixel shuffle operation (it's conditional, but should be the default), that from what I understand from glimpsing over the article, should not be affected by the issue.

[JunbinWang]

I'm not sure though if this entirely applies to the models used here, as they use a pixel shuffle operation (it's conditional, but should be the default), that from what I understand from glimpsing over the article, should not be affected by the issue.

Yeah，I check your code again carefully today. We did apply pixel shuffle in the RRDN model.
So do you have any idea about what caused the above artifacts ... ?

[cfrancesco]

I mean, I could easily be wrong and that could also be an issue with the pixel shuffling operation. You should try to experiment with strides and kernel sizes. Let me know how it goes.
Another suggestion is to start with a purely MAE driven learning. This should avoid those very evident artefacts. Then you continue with gans

[Dogdog00]

Dear all,

When I try to train a ArtefactCancelling ESRGAN model, I get some result images with strange patterns.

I put some sample images here:

I first train the model with only mae loss as the script below.

lr_train_patch_size = 50

layers_to_extract = [5, 9]



scale = 3

hr_train_patch_size = lr_train_patch_size * scale



rrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':32, 'G0':32, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)



from ISR.train import Trainer

loss_weights = {

  'generator': 1,

  'feature_extractor': 0.0,

  'discriminator': 0.0

}

losses = {

  'generator': 'mae',

  'feature_extractor': 'mse',

  'discriminator': 'binary_crossentropy'

}



log_dirs = {'logs': './logs', 'weights': './weights'}



learning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}



flatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}



trainer = Trainer(

    generator=rrdn,

    discriminator=None,

    feature_extractor=None,

    lr_train_dir='train_LR',

    hr_train_dir= 'train_HR',

    lr_valid_dir='valid_LR',

    hr_valid_dir='valid_HR',

    loss_weights=loss_weights,

    learning_rate=learning_rate,

    flatness=flatness,

    dataname='hotel',

    log_dirs=log_dirs,

    weights_generator=None,

    weights_discriminator=None,

    n_validation=40,

)



trainer.train(

    epochs=100,

    steps_per_epoch=700,

    batch_size=16,

    monitored_metrics={'val_generator_PSNR_Y': 'max','val_generator_loss': 'min','val_loss': 'min'}

)

Then I train the model with the following configuration ：

lr_train_patch_size = 50

layers_to_extract = [5, 9]



scale = 3

hr_train_patch_size = lr_train_patch_size * scale



rrdn  = RRDN(arch_params={'C':4, 'D':3, 'G':32, 'G0':32, 'T':10, 'x':scale}, patch_size=lr_train_patch_size)

f_ext = Cut_VGG19(patch_size=hr_train_patch_size, layers_to_extract=layers_to_extract)

discr = Discriminator(patch_size=hr_train_patch_size, kernel_size=3)



from ISR.train import Trainer

loss_weights = {

  'generator': 0.1,

  'feature_extractor': 0.8,

  'discriminator': 0.1

}

losses = {

  'generator': 'mae',

  'feature_extractor': 'mse',

  'discriminator': 'binary_crossentropy'

}



log_dirs = {'logs': './logs', 'weights': './weights'}



learning_rate = {'initial_value': 0.0004, 'decay_factor': 0.5, 'decay_frequency': 30}



flatness = {'min': 0.0, 'max': 0.15, 'increase': 0.01, 'increase_frequency': 5}



trainer = Trainer(

    generator=rrdn,

    discriminator=discr,

    feature_extractor=f_ext,

    lr_train_dir='train_LR',

    hr_train_dir= 'train_HR',

    lr_valid_dir='valid_LR',

    hr_valid_dir='valid_HR',

    loss_weights=loss_weights,

    learning_rate=learning_rate,

    flatness=flatness,

    dataname='hotel',

    log_dirs=log_dirs,

    weights_generator='weights/rrdn-C4-D3-G32-G032-T10-x3/rrdn-C4-D3-G32-G032-T10-x3_perceptual_epoch099.hdf5',

    weights_discriminator=None,

    n_validation=40,

)



trainer.train(

    epochs=400,

    steps_per_epoch=500,

    batch_size=16,

    monitored_metrics={'val_generator_loss': 'min', 'val_generator_PSNR_Y': 'max','val_loss': 'min'}

)

Can somebody give me some hints about what may cause the unpleased image patterns?

Best Regards,

Even

How do you determine the weight of these three parameters? What do the three of them stand for? How to adjust to achieve optimal? loss_weights = {

'generator': 0.1,

'feature_extractor': 0.8,

'discriminator': 0.1