TSL: Transitioning default GLSL Matrices to TSL

[Spiri0]

Description

In webGL you only need to declare the standard matrices in a RawShader, but you don't have to initialize or update them by yourself using the uniforms. Threejs does this automatically. I have two simple shaders here as an example in webGL and webGPU. The four standard matrices can be seen in both.

//rawShader glsl shader
const vertexShader = `
	precision highp float;
	
	uniform mat4 viewMatrix;	
	uniform mat4 modelMatrix;
	uniform mat4 projectionMatrix;
	uniform mat4 modelViewMatrix;
			
	in vec3 position;
	
	void main(){
		gl_Position = projectionMatrix * viewMatrix * modelMatrix * vec4(position, 1.0);			
	}
`;

//vertexNode wgsl shader
const vertexShader = wgslFn(`
fn mainFunction(
	position: vec3<f32>,
	viewMatrix: mat4x4<f32>,
	modelMatrix: mat4x4<f32>,
	projectionMatrix: mat4x4<f32>,
	modelViewMatrix: mat4x4<f32>,
) -> vec4<f32> {
     
	return projectionMatrix * viewMatrix * modelMatrix * vec4<f32>(position, 1.0);
}`);

For the wgslFn and tslFn you have to initialize and update these four matrices yourself. It would be nice if this would also happen automatically like in a RawShader, because the effort can be considerable if you have to do it yourself with a more complex app

Solution

The solution is to adopt the functionality of threejs for the rawShaders from webGL into the wgslFn and tslFn.

Alternatives

At the moment this can be done through manual initialization and updates via the material parameters of a MeshBasicNodeMaterial.

Additional context

No response

[Spiri0]

I found a manual way to do this, but it's pretty complicated and not practical for the average user. The problem is the modelMatrix because it is usually different for one material but different meshes. Threejs seems to create its own material instance internally in webgl for each mesh in order to be able to use the individual modelMatrix.
I was able to solve this with attributes in webgpu and then calculate the individual modelMatrix in the shader, but that is very laborious.
That this is done automatically in webgl by threejs by just declaring the matrices is a huge relief. I didn't think this would be such a complex topic, but it certainly is.

[Spiri0]

@sunag, is there anything I can do to help here?
I started to look at how it works in the gl code in three.module.js with the RawShaders. I'm far from getting through it yet, but I've realized that MeshBasicNodeMaterial basically seems to get everything it needs from the Material class. Without my own shader if I only use the colorNode then new meshes that I add to a group are always supplied with the correct, their own modelMatrix, even if I only have one MeshBasicNodeMaterial. Apparently a lot of things are already working almost to their goal.
But I didn't recognize where the MeshBasicNodeMaterial forwards everything to a final shader.

[sunag]

In case of using an object with default parameters, could it help?

import { modelViewMatrix, cameraProjectionMatrix, ... }

const defaultParameters = { modelViewMatrix, projectionMatrix: cameraProjectionMatrix, ... };

material.vertexNode = someNodeFunction( { otherParameter, ... defaultParameters } );

[Spiri0]

I tested your suggestion, but I had to create a new export in the ModelNode with modelMatrix analogous to modelViewMatrix because there was no modelMatrix node.
Unfortunately it doesn't work that way. I can see an image but when I move the transformations in wgsl are unfortunately not correct. To be honest, I didn't expect that, because if you supply the shader with matrices externally, it's not the same as in GLSL, where the RawShader uses the individual updated model matrix for each object. If I try to pass the respective model matrix externally in glsl, i.e. via the uniforms, the shader only uses one and then the transformations are not correct either. In this way I can reproduce the negative behavior in glsl like in wgsl 1:1.

I've already thought about how I can make the simplest example possible, because the code that I want to port from webgl to webgpu is very large. The nice thing is that up to this point everything works exactly the same with webgpu. From this point onwards the differentiation occurs. I have the two test shaders here as an illustration. With exactly the glsl code the representation is correct. With the wgsl code below in the webgpu variant, the code runs but unfortunately the transformations are not correct.

//---------------------------glsl----------------------------------

createMaterial(params){

	const vertexShader = `
		in vec3 position;
		uniform mat4 viewMatrix;
		uniform mat4 modelMatrix;
		uniform mat4 projectionMatrix;
	
		void main(){
	
			mat4 originViewMatrix = mat4(
				viewMatrix[0],
				viewMatrix[1],
				viewMatrix[2],
				vec4(0.0, 0.0, 0.0, 1.0)
			);
	
			gl_Position = projectionMatrix * originViewMatrix * modelMatrix * vec4(position, 1.0);
		}
	`;
	
	
	const fragmentShader = `
		precision highp float;
		out vec4 fragColor;

		void main() {
			fragColor = vec4(1.0, 0.0, 0.0, 1.0);
		}
	`;


	this.material = new THREE.RawShaderMaterial({
		glslVersion: THREE.GLSL3,
		uniforms: {},
		vertexShader: vertexShader,
		fragmentShader: fragmentShader,
		wireframe: true	
	});
}

//---------------------------wgsl----------------------------------

createMaterial(params){

	const vertexShader = wgslFn(`
		fn main_(
			position: vec3<f32>,
			viewMatrix: mat4x4<f32>,
			modelMatrix: mat4x4<f32>,
			projectionMatrix: mat4x4<f32>,
		) -> vec4<f32> {
				
			var originViewMatrix = mat4x4f(
				viewMatrix[0],
				viewMatrix[1],
				viewMatrix[2],
				vec4f(0.0, 0.0, 0.0, 1.0)
			);
		
			var outputPosition = projectionMatrix * originViewMatrix * modelMatrix * vec4<f32>(position, 1.0);

			return outputPosition;
		}
	`);
	
		
	const shaderParams = {
		position: attribute("position"),
		viewMatrix: params.camera.matrixWorldInverse,
		modelMatrix: modelMatrix,
		projectionMatrix: cameraProjectionMatrix
	}

	this.material = new MeshBasicNodeMaterial();
	this.material.vertexNode = vertexShader(shaderParams);
	this.material.colorNode = vec4(1, 0, 0, 1);
	this.material.wireframe = true;
}

I tried to trace the path of the matrices through the material inheritance to the RawShader in three.module.js.
The RawShader simply takes over from the BasicMaterial and this from the material. The renderer and the material system pass all the individual model matrices through to the RawShader. And so also to the MeshBasicNodeMaterial. But I don't recognize how the RawShaderMaterial brings the individual model matrices into the shader. In the RawShader you just need to declare the matrices and the RawShaderMaterial then knows how to supply them.

My thought was to try to reproduce exactly that for tsl and wgsl shader, because the necessary information is somewhere in the MeshBasicNodeMaterial as well as in the RawShaderMaterial. The MeshBasicNodeMaterial also uses the individual modelMatrices for all objects because otherwise the individual meshes would not be initialized correctly. But fortunately that is the case. And I find that all very reassuring.

But without access to the individual modelMatrices in the shader I cannot implement the transformation. This is part of a floating point origin transform that I use to set the camera as the origin.
It couldn't be easier than simply declaring the matrices in TSL and WGSL. The RawShader has become pretty good in that regard. Seems like internal matrix nodes in webgl.

[sunag]

I didn't expect that, because if you supply the shader with matrices externally, it's not the same as in GLSL, where the RawShader uses the individual updated model matrix for each object.

This should happen, otherwise the examples wouldn't work correctly. I believe it is related to the nodes used as input.
Nodes with the prefix model* it will use the individual updated matrices for each object.

A basic MVP should be:

import { cameraProjectionMatrix, modelViewMatrix, positionLocal } from 'three/nodes'

material.vertexNode = cameraProjectionMatrix.mul( modelViewMatrix ).mul( positionLocal );

[Spiri0]

Ah, I think I understand. I also passed the modelViewMatrix to the shader and then did the usual "cameraProjectionMatrix * modelViewMatrix * vec4f(position, 1.0)" in the shader.
And your tsl version with which I see exactly the same thing. So the individual modelViewMatrices are used.

Then my naive thought to insert this here in ModelNodes.js and adding modelMatrix on line 93 in Nodes.js was not that bad at all.

export const modelMatrix = nodeImmuable( ModelNode, ModelNode.MODEL_MATRIX ).label( 'modelMatrix' ).temp( 'ModelMatrix' );

With the viewMatrix from the camera, I get the same result.

So now I know a lot more thanks to your explanation and a lot of it fits together. But why the transformation still doesn't work in contrast to my analog code with my RawShader is still a mystery to me at the moment. I must have overlooked something with the modelMatrix, because the modelMatrixes are not updated correctly. If instead of my self made modelMatrix node I simply use a mesh.matrixWorld from one of the meshes and thus update the uniform at every interval then my origin transformation works wonderfully. All meshes move smoothly. The transformation is correct, but of course the mesh positions are not correct because I use the same mesh matrix for each mesh. I now suspect that my simple attempt to add a modelMatrix node is incomplete.

[sunag]

For world matrix the node modelWorldMatrix should work?

[Spiri0]

Yes, that works 😄
I was so focused on exclusion criteria to narrow down the cause that I didn't think of using the modelWorldMatrix node from ModelNode.js to test. I looked at the module for the first time yesterday after you mentioned the matrix nodes.

The fact that with the modelWorldMatrix looks exactly right is due to my coordinate transformation mechanism. It would therefore make sense to include the modelMatrix in ModelNode.js. I can test that then.

A viewMatrix node would round off the range. Personally, I'm fine with simply accessing the camera's matrixWorldInverse, but with a viewMatrix node you would have pretty much all the matrices together. And with it all possibilities

[sunag]

The viewMatrix is called cameraViewMatrix

[Spiri0]

I'll rename the issue to "Add modelMatrix to ModelNode.js"
This is more accurate now

Your answer in the forum to my post-processing question helped me a lot.

I noticed something interesting in your example that you recommended to me that could perhaps elegantly close an older issue, but I still have to take a closer look at it.

[sunag]

Nice! Why modelMatrix, do you think in use local matrix?

[Spiri0]

Yes, local matrix, that's the only one missing from the ModelNode.js range. But since only a few users work with matrices, this is nothing urgent

[sunag]

I think modelMatrix is confuse for a local matrix it's because modelMatrix is even used as world matrix in GLSL I think modelLocalMatrix sounds better, it would be cool to have some real use cases too.

[sunag]

Comparisons between GLSL and TSL default properties.

Type	GLSL	TSL
Attribute Vec3	position	positionLocal
Uniform Mat4	viewMatrix	cameraViewMatrix
Uniform Mat4	modelMatrix	modelWorldMatrix
Uniform Mat4	modelViewMatrix	modelViewMatrix
Uniform Mat4	projectionMatrix	cameraProjectionMatrix

@Spiri0 I'm going to close the topic as resolved and rename the title, if you think it's necessary you can open another one about modelLocalMatrix, the conversion flow is about naming, I think it would help other people with similar problems who search on Google or Github.

[Spiri0]

I was just thinking about one of my previous apps where I often use the local matrix, but that was always only on the javascript side in the 3js code and never in the shader.
Yes, you are right, the issue can then be closed