Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Anime4K_Upscale_GAN_x2_M seem to be producing some minor artifacts that are not present in the other GAN variants #146

Open
Z-Dante opened this issue Nov 29, 2021 · 9 comments
Assignees

Comments

@Z-Dante
Copy link

Z-Dante commented Nov 29, 2021

Not really sure if this is a bug or not, but earlier I was playing with the new GAN shaders and noticed one thing. The X2_M variant of the GAN shader seem to be producing some artifacts here and there, which are not present in any other GAN shaders. To demonstrate, here are some images.

  1. Some noisy old gameplay record 480p->1080p: Zoom in and look at the border of the buttons if you can't notice the artifacts
    image
  2. 720p -> 1080p. 400% zoomed view:
    image
    And album of the full images for reference: https://imgur.com/a/qV66BA3

All tested with the latest windows build of MPV on default settings on a 1080p display(I don't have 4k :-x ), running on RTX2060 GPU.
I might be nitpicking but feels odd to me that only one variant of the GAN shaders would produce such artifacts.

And good work on the new shaders. These seem to be much better at keeping grainy textures intact compared to the CNN ones.

Edit: My input.conf:

CTRL+1 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders//Anime4K_Upscale_CNN_x2_L.glsl"; show-text "Anime4K: Upscale L CNN"
CTRL+2 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x2_S.glsl"; show-text "GAN X2 S"
CTRL+3 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x2_M.glsl"; show-text "GAN X2 M"
CTRL+4 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x3_L.glsl"; show-text "GAN X3 L"
CTRL+5 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x3_VL.glsl"; show-text "GAN X3 VL"
CTRL+6 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_GAN_x4_UL.glsl"; show-text "GAN X4 UL"
CTRL+7 no-osd change-list glsl-shaders set "~~/shaders/Anime4K_Clamp_Highlights.glsl;~~/shaders/Anime4K_Upscale_CNN_x2_VL.glsl"; show-text "CNN VL"
CTRL+0 no-osd change-list glsl-shaders clr ""; show-text "GLSL shaders cleared"```
@bloc97
Copy link
Owner

bloc97 commented Nov 29, 2021

Thanks for the in-depth comparison pictures! I think that it's simply by chance the M shader didn't converge correctly during training. I will try to fix this before the official release of shaders.

@bloc97 bloc97 mentioned this issue Nov 29, 2021
24 tasks
@bloc97 bloc97 self-assigned this Nov 29, 2021
@bloc97
Copy link
Owner

bloc97 commented Nov 29, 2021

In fact I think I can see very tiny (1 pixel) artifacts in the L version too. There might be broader convergence issues from the adversarial training. I will have to investigate. Solving this correctly might take more time as there's no papers that tackle training tiny SRGANs. For now I will try re-training both networks to see if the issue repeats itself.

@hooke007
Copy link

By my test, upscale GAN x2-UL could produce the best quality while the least artifacts, but it was removed. @bloc97 Could we get another choice to replace GAN x2-UL later?
x2-m x2-UL x3-L
x3-VL x4-UL


@bloc97
Copy link
Owner

bloc97 commented Nov 30, 2021

That's because GAN x2-UL was trained with small amounts of denoising, I will see if we should support 6 GAN+Denoise upscalers, or let the GAN-Restore handle denoising.

@bloc97
Copy link
Owner

bloc97 commented Nov 30, 2021

The optimal case would be to delegate all degradations to RestoreGAN shaders, and let the SRGAN work with pristine images. The current problem is that Restore shaders sometimes introduce ringing, blurs out details or do not restore lines enough.

This happens because the restore shader has no idea of the whole image and only works on small patches of the image. It cannot know if the entire image is blurred or whether the blur was intentional and only for a part of the image. This can cause the restore shader to try removing noise or blur where there is no degradation and cause a loss of detail/ringing.

A lazy method was used for the v4.0 restore shaders, by training one shader (non-soft version) on some degradation and another (soft version) on other types of degradation.

By implementing global feature-wise transformations we can hope the new RestoreGAN shaders will not denoise, deblur or add ringing to images that have no degradation.

@Z-Dante
Copy link
Author

Z-Dante commented Nov 30, 2021

I went back to check the old GAN x2-UL shader and it does seem to be working great and producing less artifacts for most cases. But it also has some edge cases where it was losing a lot of the minor details on grainy textures (not as much as the CNN shaders but still). Would be great if there was a way to reduce the artifacts while minimizing the loss of details ^^
https://imgsli.com/ODQyMjU/0/1

@bloc97
Copy link
Owner

bloc97 commented Nov 30, 2021

That's basically what we're trying to do with feature-wise transformations. If the shader can actually know how much compression is in an entire image, it can learn to not remove details when overall noise is low.

Compression and degradation is almost always applied to the entire image, while artistic degradation is usually applied to parts of an image. The important thing is to distinguish between the two by giving the neural network a way to look at the entire image without slowing it down too much.

@bloc97
Copy link
Owner

bloc97 commented Nov 30, 2021

Upon further examination of the image you provided, it is a very good example of having all kinds of artistic degradation + unintended noise from compression. There are four things:

  1. The gaussian noise (film-grain like noise) should not be removed as it is artistic and intended (you can look at the blurry parts of the image at the top and bottom, they don't have noise) Current Restore shaders would definitively try to remove it, making the image lose detail.
  2. The blur at sides of the images (vignette) is artistic and should not be removed. Current restore shaders will also try to remove it, and wrongly oversharpening parts of the image, causing ringing artifacts.
  3. There doesn't seem to have chroma subsampling, current restore shaders always assume 4:2:0 chroma subsampling and will cause color artifacts.
  4. There is JPEG/MPEG (more commonly, DCT) compression, this should be removed as usual, current restore shaders always assume a moderate amount of compression, which does the right thing here.

If you try restore shaders with this image it will fail considerably, as expected. The goal is for the new RestoreGAN shaders to take in account all four cases, so that it can make a better restoration overall.

@bloc97
Copy link
Owner

bloc97 commented Nov 30, 2021

If the RestoreGANs will be successful at restoring an image without distorting it, the SRGAN shaders will not need to denoise the image. This makes both training and using the shaders much easier, and we would not need a second set of SRGAN+Denoise shaders:

Pristine images from pixiv: Use SRGAN only.
Visual novels: Use SRGAN only.
Degraded images from web: Use RestoreGAN + SRGAN.
Anime: Use RestoreGAN + SRGAN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants