WebGPURenderer: Images with color-specific metadata produces different results in both backends.

[steve-o]

Description

REF: https://www.vantajs.com/?effect=clouds2

Taking the popular Vanta clouds2 demo and porting to TSL I can reproduce the visual output with forceWebGL: true, however when using WebGPU the clouds are significantly less fluffy. I really not sure how to start debugging this.

Reproduction steps

1. In one window: https://miffy.dsbunny.com/apps/webgpu/play/webgl.html
2. In another: https://miffy.dsbunny.com/apps/webgpu/play/webgpu.html

Code

const cloudShader = /*@__PURE__*/ Fn( ({
	resolution,
	time,
	speed,
	skyColor,
	cloudColor,
	lightColor,
	cloudTexture,
}) => {
	const coord = vec2(
		screenCoordinate.x,
		screenCoordinate.y,
	).toVar();

	// Base direction vector (0.8, 0, coord/resolution.y - 0.35)
	const d = vec4(
		float(0.8),
		float(0),
		coord.div(resolution.y).sub(float(0.35)),
	).toVar();

	// Sky gradient based on height (skyColor - d.w)
	let outColor = skyColor.sub(d.w).toVar();

	// Initial value for ray marching (200.0 + sin(dot(coord,coord)))
	let t = float(200.0).add(sin(dot(coord, coord))).toVar();

	// Ray marching loop
	const MAX_ITER = float(100.0);

	Loop({ start: float(1.0), end: MAX_ITER, type: 'float', condition: '<=' }, () => {
		// Decrease t by 2.0 for each step
		t.assign(t.sub(float(2.0)));

		// Break if t < 0
		If(t.lessThan(float(0.0)), () => { Break(); });

		// p = 0.05 * t * d
		let p = d.mul(t).mul(float(0.05)).toVar();

		// Apply movement through space
    		// p.xz += time * 0.5 * speed
		p.xz.assign(p.xz.add(time.mul(float(0.5)).mul(speed)));

		// p.x += sin(time * 0.25 * speed) * 0.25
    		p.x.assign(p.x.add(sin(time.mul(float(0.25)).mul(speed)).mul(float(0.25))));

		// Start with s = 2.0
		let s = float(2.0).toVar();

		// Initialize f value for cloud density
    		// Note: In GLSL we have p.w + 1.0 - T - T - T - T
    		let f = p.w.add(float(1.0)).toVar();

    		// Emulate -T -T -T -T
		Loop(4, () => {
			// Calculate texture coordinates for sampling
      			// fract((s*p.zw + ceil(s*p.x)) / 200.0)
			const p_zw = p.zw;
			const s_p_zw = s.mul(p_zw);
			const s_p_x = s.mul(p.x);
			const texCoord = fract(s_p_zw.add(ceil(s_p_x)).div(float(200.0)));

			const sampleY = texture(cloudTexture, texCoord).y.div(s.assign(s.add(s)));
			const T = sampleY.mul(float(4.0));

			// Subtract T from f (f -= T)
      			f.assign(f.sub(T));
		});

		// Check if inside a cloud (f < 0.0)
		If(f.lessThan(float(0.0)), () => {
			// Calculate cloud color with shading
      			// mix(lightColor, cloudColor, -f)
			const shaded = mix(lightColor, cloudColor, f.negate());

			// Blend with the sky color
      			// mix(out1, cloudColorShading, -f * 0.4)
			const mixed = mix(outColor, shaded, f.negate().mul(float(0.4)));
			outColor.assign(mixed);
		});
	});

	// Return final color with alpha = 1.0
	return vec4(outColor, float(1.0));
} );

Live example

• jsfiddle-latest-release WebGPURenderer

Screenshots

Version

176

Device

Desktop

Browser

Chrome

OS

Windows

[Mugen87]

Updated live example: https://jsfiddle.net/rw2bnjvq/

[Mugen87]

The root cause seems to be the transparent PNG noise texture which is sampled differently in both backends. Using a JPG without transparency and both backends render the same: https://jsfiddle.net/7quds5nk/

Below are two live examples that isolate the issue. The code samples the noise texture (PNG) and displays the green value as a grayscale with constant alpha (1).

WebGPU: https://jsfiddle.net/c5tdLm63/
WebGL: https://jsfiddle.net/c5tdLm63/1/

The WebGL backend renders as expected like WebGLRenderer so the issue must be located somewhere in the WebGPU backend.

[Mugen87]

In Safari with enabled WebGPU, both fiddles render identical. Only in Chrome there is a difference.

Could this be a browser issue and Chrome's WebGPU implementation does not sample the semi-transparent color values correctly?

[steve-o]

Can raise with the Chome team and get their feedback? Thanks for updating the fiddles.

Chromium ticket: https://issues.chromium.org/issues/418746324

[Mugen87]

@greggman Do you think we see a potential Chromium issue here or do we overlook something WebGPU specific?

The issue is summed up in #31132 (comment). I've made sure the reproduction test cases do not apply any color transformation in the shaders (so no tone mapping or color space conversion).

[WestLangley]

This appears to be related to premultiplied alpha. These two test cases match on chrome (only).

WebGL: https://jsfiddle.net/3m7js4ry/

material.colorNode = vec4( color.r.mul( color.a ), color.g.mul( color.a ), color.b.mul( color.a ), 1 );

//

WebGPU: https://jsfiddle.net/9fjn4d2h/

material.colorNode = vec4( color.r, color.g, color.b, 1 );

[steve-o]

I think the Chromium ticket update is saying invalid API usage.

[greggman]

I think the Chromium ticket update is saying invalid API usage.

I don't think that's the case yet. Still checking...

[greggman]

I guess the short version for this exact problem is, replace that noise.png with a new one. The issue seems to be that noise.png has gAMA and cHRM chunks. Those are applied or ignored differently in WebGL vs WebGPU, partly as specced. IIUC, the workaround to ignore those chunks when using WebGPU is to load the image via createImageBitmap and pass in { colorSpaceConversion: 'none' }. Unfortunately, that path in all browsers still appears to apply at least one of those chunks differently in WebGPU vs WebGL so it's not currently a solution. Hopefully that can be fixed.

Another solution would be to decode the images yourself 🤮 Then you can chose to apply or not apply color space. I'm not recommending it but, mentioning it since getting browsers to handle this stuff consistently is difficult. The good thing is deflate decompression is now JavaScript API so it might not be that much code to write a PNG loader.

ps: still need to find why things are different in Safari for the examples above

[Mugen87]

Thank you for this valuable feedback! It's somewhat a relief the issue is so good understood and it is not a bug in the WebGPU implementation or on our side.

It's actually not the first time that such metadata result in unexpected behavior. For example #30471 demonstrates how browsers evaluate rotation metadata in MPEG files in different ways leading to inconsistent behavior of VideoTexture. The best solution so far is to encode the MPEG without such metadata.

It's true that with custom image loaders the engine would have more control over the image data. On the other side this would add additional maintenance burdens. Besides, developers can load image data without one of our loaders (ImageLoader, ImageBitmapLoader or TextureLoader) and then we have the same issue again. I personally would prefer to rely on the Web API for such tasks.

Regarding the discussion at the Chromium bug: I like the suggestion of a consistent behavior of createImageBitmap() across WebGL and WebGPU. In the long term ImageBitmapLoader becomes hopefully the default image loader in the engine. In context of glTF, the respective importer already uses it by default when support is detected. Next to the improved runtime behavior, are more consistent result when processing images would be an additional argument for using image bitmaps.