Upscaling & Denoise filters causes browser sources to lose transparency

[HunterAP23]

Operating System

Windows 10 1903 and higher

OBS Studio Version?

27.1

StreamFX Version

0.11.0a4

OBS Studio Log

https://github.com/Xaymar/obs-StreamFX/files/7289391/2021-10-05.14-48-08.txt

OBS Studio Crash Log

No response

Current Behavior

When using the new Upscaling or Denoise filters on Browser sources, the Browser source loses any existing transparency.

This was moved from #642 as this issue is related to but is different from that issue.

Expected Behavior

Browser sources should not lose any of their transparency when applying an Upscaling or a Denoise filter to them.

Steps to Reproduce the Bug

1. Add a browser source that contains transparency.
2. Apply either an Upscaling or Denoising filter to the Browser source in question.

Any additional Information we need to know?

The Denoise filter always removes the Browser source's transparency, regardless of it's settings.

The Upscaling filter strangely only causes this issue when the "Scale" option is set to 141.67% or higher - anything lower and the issue is not present. Similarly to what happens in #642 , the Browser source will shift in size, but only when the "Scale" option is set to 141.67% or higher. This seems to be more of an issue where the upscaling does not occur until a specific threshold is reached, as the size of the Browser source updates again when hitting a Scale value of 175.01%, and again at 250.01%.

My understanding is that since the Nvidia upscaling technology upscales in specific intervals (1.5x, 2x, 2.5x, etc.) and StreamFX only upscales the source when it reaches those multiples.

[Xaymar]

I'm unsure if I want to treat it as a bug so far, since these effects don't scale transparency as well as color, so the transparency channel is quite literally undefined. This is true for every single upscaling algorithm I know, none of them consider transparency in an image. So what would the expected result here be, since we have nothing to scale the transparency with? Just simply interpolate with linear sampling?

My understanding is that since the Nvidia upscaling technology upscales in specific intervals (1.5x, 2x, 2.5x, etc.) and StreamFX only upscales the source when it reaches those multiples.

You are correct that the underlying NVIDIA Super Resolution network only supports certain scale factors, to which StreamFX snaps. Any unsupported resolution will cause the network to fail initialization, which results in the out-of-bounds rendering.

[HunterAP23]

I'm unsure if I want to treat it as a bug so far, since these effects don't scale transparency as well as color, so the transparency channel is quite literally undefined. This is true for every single upscaling algorithm I know, none of them consider transparency in an image. So what would the expected result here be, since we have nothing to scale the transparency with? Just simply interpolate with linear sampling?

Linear sampling could work, or the more complicated (and probably unnecessary way) would be to make it take the alpha channel, run it through a separate denoising instance, then apply it back as an alpha channel to the original source's output.
I can only imagine this helps with something like pre-rendered video media sources like overlays and such, which are likely to have an alpha channel for transparency, and that transparency may be relatively complicated for the given source.

You are correct that the underlying NVIDIA Super Resolution network only supports certain scale factors, to which StreamFX snaps. Any unsupported resolution will cause the network to fail initialization, which results in the out-of-bounds rendering.

That makes sense. I'm not sure if there's a worthy tradeoff to making so the source is upscaled to the next available multiplying factor, then downscale it back to the defined percentage.
IE: if the user specified a scaling factor of 140%, then the filter would upscale using the nearest scaling factor (which I believe would be 1.5x or 150%), then downscale the source back to fit into that 140% factor. With a given source at 1080p resolution, a 150% upscale would make it 2880x1620. Then filter can apply downscaling down to 140% of the resolution, which is 2688x1512.

The "how worth it is to spend resources upscaling and then downscaling again" thing is definitely an issue, although I presume in some scenarios it's worth it, while in others it's not.

[Xaymar]

That makes sense. I'm not sure if there's a worthy tradeoff to making so the source is upscaled to the next available multiplying factor, then downscale it back to the defined percentage.

The problem with that is that there is no defined list of resolutions explicitly supported, and the network fails if it's off by as little as 1px in any direction. Which is a bit difficult to detect without reloading the effect, and that takes a fraction of a second - so it would introduce a noticable hitch every time you move the slider.