The Behavior of RenderFlusher

[panxinmiao]

I've noticed before that compared to other engines, when rendering the same scene, pygfx sometimes appears "slightly blurry" in the rendering result. This is not very noticeable in most cases.

However, in my recent use of imgui (integrated with wgpu), I found that the imgui elements (especially fonts) appeared very blurry when rendering.

After some investigation, it seems that this is caused by the behavior of pygfx's RenderFlusher.

pygfx/pygfx/renderers/wgpu/engine/flusher.py

Lines 253 to 270 in eb0b089

	// Convolve. Here we apply a Gaussian kernel, the weight is calculated
	// for each pixel individually based on the distance to the ref_index.
	// This means that the textures don't need to align.
	var val: vec4<f32> = vec4<f32>(0.0, 0.0, 0.0, 0.0);
	var weight: f32 = 0.0;
	for (var y:i32 = -support; y <= support; y = y + 1) {
	for (var x:i32 = -support; x <= support; x = x + 1) {
	let step = vec2<i32>(x, y);
	let index = clamp(base_index + step, min_index, max_index);
	let dist = length(ref_index - vec2<f32>(index) - 0.5);
	let t = dist / sigma;
	let w = exp(-0.5 * t * t);
	val = val + textureLoad(r_color, index, 0) * w;
	weight = weight + w;
	}
	}
	let gamma3 = vec3<f32>(u_render.gamma);
	out.color = vec4<f32>(pow(val.rgb / weight, gamma3), val.a / weight);

If I understand correctly, the logic here is to implement the MSAA algorithm in the shader? This seems to be causing the blurriness in the result.

As a comparison, the following are the results of not using RenderFlusher's current behavior:

Additionally, wgpu can support MSAA in hardware. Is there any special consideration behind the logic of RenderFlusher here?

[panxinmiao]

The contrast is more evident in this example:

[almarklein]

That code does SSAA. By default everything is rendered with a pixel_ratio of 2, and when the offscreen buffer is rendered to screen there's two possible cases:

• The pixels match exactly, because you're on a hidpi screen.
• The screen has less pixels than the offscreen buffer: filter with ssaa.
• The screen has more pixels. Unlikely scenario.

This triage happens here:

pygfx/pygfx/renderers/wgpu/engine/flusher.py

Lines 177 to 185 in eb0b089

	if factor == 1:
	# With equal res, we smooth a tiny bit. A bit less crisp, but also less flicker.
	ref_sigma = 0.5
	elif factor > 1:
	# With src a higher res, we will do SSAA.
	ref_sigma = 0.5 * factor
	else:
	# With src a lower res, the output is interpolated. But we also smooth to reduce blockiness.
	ref_sigma = 0.5

I've tweaked the parameters at the time for visually appealing results, but it can well be that for text a more crisp result is desired. So we may want to re-tweak the ref_sigma in the code above. This should be done separately for both hidpi and non-hidpi screens.

BTW, the filter strength can be set with the WgpuRenderer.pixel_filter param (can be a float).

[almarklein]

Out of interest, how are you using wgpu with imgui? I think an example that uses imgui would be very interesting since a lot of downstream projects are eager for solutions to have simple gui elements ...

[panxinmiao]

Out of interest, how are you using wgpu with imgui? I think an example that uses imgui would be very interesting since a lot of downstream projects are eager for solutions to have simple gui elements ...

I am using the library imgui-bundle, which provides Python bindings for imgui.
Referring to the code at opengl_backend.py , I have implemented a wgpu (wgpu-py) backend.

After refactoring and optimizing the code, I will submit a PR.
Ultimately, I hope to provide a convenient component similar to lil-gui, but for the Python ecosystem within pygfx.

I am considering how to encapsulate the API, and I may submit separate PRs to the wgpu-py and pygfx libraries.

[panxinmiao]

That code does SSAA. By default everything is rendered with a pixel_ratio of 2.

I thought that as long as we rendered the scene at a higher resolution and then downsampled (using a linear filtering sampler), it would be considered SSAA, 😅

BTW, the filter strength can be set with the WgpuRenderer.pixel_filter param (can be a float).

I tried setting the pixel_filter to 0, and the text did become sharper, but the scene aliasing was quite noticeable.

Can we provide an option in the Renderer that allows users to disable the current behavior of the RenderFlusher (directly using a linear-filtering sampler instead)?

Additionally, we could provide an option to enable MSAA (hardware acceleration).

Regarding the API design, I have the following two suggestions:

• Option 1, two params:

  • ssaa (bool):

    When True, use the current behavior;
    When False, directly use a linear-filtering sampler.

  • msaa (bool):

    When True, enable hardware-accelerated MSAA;
    When False, do not enable.

• Option 2, one param:

  • antialias (enum): Can be "MSAA", "SSAA", or None, (perhaps an additional "Linear-filtering" option).

    When set to "SSAA", use the current logic.
    When set to "MSAA", use hardware-accelerated MSAA.
    When set to None, do not apply anti-aliasing, or directly use a linear filtering sampler.

If you agree, I can submit a PR.

[almarklein]

I think the first option makes the most sense, since you can have both ssaa and msaa at the same time.

That code does SSAA. By default everything is rendered with a pixel_ratio of 2.

I thought that as long as we rendered the scene at a higher resolution and then downsampled (using a linear filtering sampler), it would be considered SSAA, 😅

It kindof is. Except if the screen is hidpi (and the pixel ratio already matches) we can just blit the pixels, so its not saa, really. We've now opted for a tiny bit of smoothing in this case, but we can remove/reduce that.

Linear filtering is technically not an appropriate kernel to downsample data. It now uses a Gaussian filter, but now that I'm thinking about this I recall this is also not ideal - especially for small sigmas - because the data is discrete. I will look into some old code that implements Lindeberg's diffusion kernel. Apart from that, we could tune the kernel down a bit.

The current behavior when the pixel_filter is zero is nearest interpolation, but perhaps it should be linear at the least.

And we can do #75 :)

If you want to look into msaa, I can have a look at improving the "ssaa" filtering.

[panxinmiao]

If you want to look into msaa, I can have a look at improving the "ssaa" filtering.

Okay, I will try to add MSAA.