Color shift with srgan model

[fjallraven]

Hello

First of all, thank you for your work on this.
I've noticed a slight color shift on inference from srgan model.
Tried tweaking and normalize methods to modify with div2k mean values and retrain the mode, but did not improve the results.

Thank you

[krasserm]

Can you please be more specific?

• how did you run inference?
• share sample images?
• ...

[fjallraven]

I've attached my result for srgan and edsr plus manual upres image to compare. You can best see a slight shift in the background behind the cat.
It's tricky to spot side by side, but if you flip between them you can see srgan is slightly more red.

Using the pretrained weights you have provided.
Code is below images.

manual resize:

edsr, overall same color values as above:

srgan, slightly more red (comparing background against edsr and manual resized above)

srgan:

from model.srgan import sr_resnet
from model.common import resolve
from utils import load_image
import tensorflow as tf
import numpy as np
import scipy.misc
model = sr_resnet()
model.load_weights('weights/srgan/gan_generator.h5')
lr = load_image('demo/0869x4-crop.png')
lr = lr[:,:,:3]
h, w, c = lr.shape
lr = lr.reshape(1, h, w, 3)
sr = resolve(model, lr)
sr = np.array(sr)
e, h, w, c = sr.shape
im = np.zeros((w, h, 3))
sr = sr.reshape((h, w, 3))
scipy.misc.imsave('result/srgan_test.png', sr)

edsr:

from model.edsr import edsr
from model.common import resolve
from utils import load_image
import tensorflow as tf
import numpy as np
import scipy.misc
model = edsr(num_res_blocks=16)
model.load_weights('weights/edsr-16-x4/weights.h5')
lr = load_image('demo/0869x4-crop.png')
lr = lr[:,:,:3]
h, w, c = lr.shape
lr = lr.reshape(1, h, w, 3)
sr = resolve(model, lr)
sr = np.array(sr)
e, h, w, c = sr.shape
im = np.zeros((w, h, 3))
sr = sr.reshape((h, w, 3))
scipy.misc.imsave('result/edsr_test.png', sr)

[krasserm]

Honestly, I cannot see a color shift by plain visual inspection. I asked for some samples to make sure there are no gross distortions but from what you provided these do not exist. Can you quantify this red-shift and tell if it is significant?

[fjallraven]

Having tested lot of more images - I will close this issue.
Just a small visual difference, and it really depends on the input. But it's probably just down to the more fine texture detail compared to edsr

Thank you

[devernay]

I had the same issue with https://github.com/Tencent/Real-SR so I guess that color shift is not (yet) considered a real issue in SR methods but the community (only PSNR takes into account color shift, SSIM and LPIPS consider it a minor disturbance).

Here’s how I remove the color-shift from super-resolved images: downscale the SR image, compute the difference with the LR image, Gaussian blur, upscale, add to the SR image. All computation is done in linear color space.
This gives this one-line G'MIC script (edited to work also with older versions of gmic):

gmic image.jpg image_4x.jpg -srgb2rgb [1] -resize[1] 25%,25%,[0],[0],2 -sub[0,1] -blur[0] 2 -resize[0] 400%,400%,[0],[0],3 -add -rgb2srgb -output image_4x_nocolorshift.jpg

[krasserm]

Thanks for the useful comment @devernay. Wasn't aware of that. Can you quantify the impact of this correction has on PSNR?

[devernay]

@krasserm sorry no, I didn't measure that, as I was mainly interested by the visual quality, and the color difference between SR-then-downscaled and the original image in my case was too large to get unnoticed (I first noticed it on vegetation and on various textured textiles).
I noticed it on Tencent's Real-SR, which won a SR challenge this year, and then did some research for mentions of color-shift, and only found this github issue concerning SRGAN - which is not very far from Tencent's Real-SR.
Note that the fix posted above does a local average of the color shift and subtracts it from the SR image, which does not fix everything.
I've seen Real-SR adding a red tint to black-and-white textures, and fixing it means adding some cyan to the reddish white, but also adds a bit of cyan to the black. So in terms of PSNR it should be better, but there is still room for improvement, for example by modeling a color-dependent local color shift, rather than a locally constant color shift.

Why this happens still escapes me, and I haven't looked into details, but if the training data is generated by processing gamma-compressed (sRGB) pixel values rather than linear colors, that may be an explanation.

Any image resizing, blurring, convolution, etc should be done in linear color space (Eric Brasseur has the best explanation on the topic, and Helmut Dersch's explanation also helps). You will notice in my G'MIC command-line above that srgb2rgb is the first thing I do, and rgb2srgb is the last thing. All processing is done in floating-point arithmetic.

The deep network could "learn" to do the SR properly (in linear rather than gamma space), but it still needs to have the right training data to do so.