VTF morph targeting with multiple meshes sharing same position/normal attributes

[0b5vr]

Describe the bug

Context: I'm a contributor of VRM, which is a humanoid-oriented glTF extension which is intended to be used as humanoid avatars.

I noticed vram consumptions of glTF models are increasing starting from Three.js r133, sometimes loading single humanoid glTF is enough to knock-off my WebGL context.
I compared between r132 and r133 using greggman/webgl-memory and and I spotted an unforgivable difference at the texture memory consumption between them.
(I can't provide the model I used for the experiment since reasons, I'm sorry)

r133 has a big improvement on morph targeting part, enabling using 8 or more morph targets at the same time using VTF morph targeting (I mean, I appreciate this change a lot).
#22293

I've looked into the code behind the VTF morph targeting, and I noticed that, despite the model shares a single position/normal attributes between meshes the morph target textures are generated for each meshes.

three.js/src/renderers/webgl/WebGLMorphtargets.js

Lines 53 to 156 in 1a241ef

	// instead of using attributes, the WebGL 2 code path encodes morph targets
	// into an array of data textures. Each layer represents a single morph target.
	const numberOfMorphTargets = geometry.morphAttributes.position.length;
	let entry = morphTextures.get( geometry );
	if ( entry === undefined || entry.count !== numberOfMorphTargets ) {
	if ( entry !== undefined ) entry.texture.dispose();
	const hasMorphNormals = geometry.morphAttributes.normal !== undefined;
	const morphTargets = geometry.morphAttributes.position;
	const morphNormals = geometry.morphAttributes.normal || [];
	const numberOfVertices = geometry.attributes.position.count;
	const numberOfVertexData = ( hasMorphNormals === true ) ? 2 : 1; // (v,n) vs. (v)
	let width = numberOfVertices * numberOfVertexData;
	let height = 1;
	if ( width > capabilities.maxTextureSize ) {
	height = Math.ceil( width / capabilities.maxTextureSize );
	width = capabilities.maxTextureSize;
	}
	const buffer = new Float32Array( width * height * 4 * numberOfMorphTargets );
	const texture = new DataTexture2DArray( buffer, width, height, numberOfMorphTargets );
	texture.format = RGBAFormat; // using RGBA since RGB might be emulated (and is thus slower)
	texture.type = FloatType;
	texture.needsUpdate = true;
	// fill buffer
	const vertexDataStride = numberOfVertexData * 4;
	for ( let i = 0; i < numberOfMorphTargets; i ++ ) {
	const morphTarget = morphTargets[ i ];
	const morphNormal = morphNormals[ i ];
	const offset = width * height * 4 * i;
	for ( let j = 0; j < morphTarget.count; j ++ ) {
	morph.fromBufferAttribute( morphTarget, j );
	if ( morphTarget.normalized === true ) denormalize( morph, morphTarget );
	const stride = j * vertexDataStride;
	buffer[ offset + stride + 0 ] = morph.x;
	buffer[ offset + stride + 1 ] = morph.y;
	buffer[ offset + stride + 2 ] = morph.z;
	buffer[ offset + stride + 3 ] = 0;
	if ( hasMorphNormals === true ) {
	morph.fromBufferAttribute( morphNormal, j );
	if ( morphNormal.normalized === true ) denormalize( morph, morphNormal );
	buffer[ offset + stride + 4 ] = morph.x;
	buffer[ offset + stride + 5 ] = morph.y;
	buffer[ offset + stride + 6 ] = morph.z;
	buffer[ offset + stride + 7 ] = 0;
	}
	}
	}
	entry = {
	count: numberOfMorphTargets,
	texture: texture,
	size: new Vector2( width, height )
	};
	morphTextures.set( geometry, entry );
	}
	//
	let morphInfluencesSum = 0;
	for ( let i = 0; i < objectInfluences.length; i ++ ) {
	morphInfluencesSum += objectInfluences[ i ];
	}
	const morphBaseInfluence = geometry.morphTargetsRelative ? 1 : 1 - morphInfluencesSum;
	program.getUniforms().setValue( gl, 'morphTargetBaseInfluence', morphBaseInfluence );
	program.getUniforms().setValue( gl, 'morphTargetInfluences', objectInfluences );
	program.getUniforms().setValue( gl, 'morphTargetsTexture', entry.texture, textures );
	program.getUniforms().setValue( gl, 'morphTargetsTextureSize', entry.size );

I can blame the structure of model itself of course, since it should have only necessary part of VBOs for each meshes plus morph targets should only be applied to necessary parts, but since the issue does not happen in r132 or prior, I want to fix this from Three.js side.

I think I can try to fix this part.

Tangential issues

I also noticed I can't free these textures when I unload the model.
Is there any way to unload the morph target textures?

Live example

https://glitch.com/edit/#!/three-r133-vtf-morph-memory-leak

Platform

• Device: Desktop
• OS: Windows
• Browser: Chrome
• Three.js version: r133, r135
  • DOES NOT REPRODUCE in WebGL1Renderer. Only happens with WebGL2.

[0b5vr]

Ok this is way harder than I thought.

Since each morph textures are an array of every morph target, we need a map from "set of morph target positions/normals" to the texture. We cannot create textures for each individual morph targets.

I was thinking about changing the key of the WeakMap morphTextures to Geometry.morphAttributes but it requires us to care about uniqueness of morphAttributes' object, which would be a dangerous design,,,
Actually models loaded via GLTFLoader has different morphAttributes objects for each geometries despite the content is identical.

I need your help @Mugen87 . First of all is this a problem that is worth resolved?
It might be better to just reconstruct VBOs and IBOs.

[Mugen87]

Why can't you share a single geometry if position and normal data are identical?

First of all is this a problem that is worth resolved?

In my opinion, no. The implementation assumes that vertex data are organized on a geometry level. It was planned to expand the data support in WebGLMorphtargets to honor colors and texture coordinates. The geometry object is the logical component that holds all these data together. Hence, one data texture per geometry.

[0b5vr]

I completely forgot to explain about the structure of the model, I'm sorry.
The model shares a same VBOs for multiple meshes while each mesh has different indices.

I hope these figures explain my circumstance well, this is how GLTF structure of the model look like:

[Mugen87]

I'm afraid we can't support this use case in WebGLMorphtargets. Consider to split up you buffers so each geometry has its unique set of (smaller) attributes.

[0b5vr]

In my opinion, no. The implementation assumes that vertex data are organized on a geometry level.

That sounds very reasonable.

I was not sure how common this kind of buffer structure is among the scene.

Consider to split up you buffers so each geometry has its unique set of (smaller) attributes.

I probably have to implement this as my next action.

Thank you for your quick response as usual!

[0b5vr]

I also noticed I can't free these textures when I unload the model.
Is there any way to unload the morph target textures?

Let me ask about this in a separated discussion.

[0b5vr]

posted: https://discourse.threejs.org/t/how-do-i-free-morph-textures/33033

[donmccurdy]

I'm afraid we can't support this use case in WebGLMorphtargets. Consider to split up you buffers so each geometry has its unique set of (smaller) attributes.

For the reasons described in #17089, having one buffer for every BufferGeometry is not as efficient as sharing buffers. I don't think we need to solve that now, but it's a pretty common layout for glTF files and if there are problems with using morph targets we'll eventually run into that again. Perhaps we can discuss it in #17089.

[0b5vr]

fyi, I've added a utility function to three-vrm, that makes morph textures compact against such models

pixiv/three-vrm#880