Tool to remove skin bind matrices from glTF file and resave?

[UltraEngine]

Is there some command-line tool that will remove the inverse bind matrices and save a glTF file, in which the initial pose is used to calculate the inverse bind matrices? I have been trying to get this to work in our software for several years, and I don't think it ever will. We can load and play the skeleton animation with no problems, and glTF files without inverse bind matrices work fine.

I understand the concept of bind matrices, but there are too many small things that can go wrong in the code. Our glTF loader collapses skinned meshes into a single top-level node, skeletons are a separate object type and hierarchy system from 3D models, and can be shared in between models. There's just too many things that can go wrong so I am looking at approaching this from the other end and feeding my loader a "clean" file.

[bghgary]

I'm guessing glTF-Transform can do it, but I'll let @donmccurdy confirm.

Since this is a question and not an issue about the spec, it would be better to use the Khronos forum to continue the conversation.

[UltraEngine]

I can confirm the glTF-transform tool "flatten" command does not remove the skin inverse bind matrices. The top-level node has a rotation of (0,0,0), but our loader still has the same issue with this model. The scale seems to remain the same as the un-collapsed mesh but that does not seem to cause any problem.

[javagl]

I'm not aware of any tool that offers this out-of-the-box. But then, the question is: Could this be implemented with "little" effort, for example, based on glTF-Transform - say, with less than 100 lines of "boilerplate" code.

I'm not sure how easy this could be. I might need a refresher from KhronosGroup/glTF-Tutorials#64 (comment) , where @emackey and @scurest explained this in more detail for me.

Right now, I'm not entirely sure whether this could be assembled from a few (pseudocode) functions like computeInverseBindMatrixForJoint and applyMatrixToMesh. I think that this is not sufficient. I think that it will be necessary to essentially "extract" the vertex positions that are acutally computed by the shader for a given set of weights (including the computation of the skinMat, but done on the CPU).

This could, in fact, be a simple solution iff there was an API to say gltf.setAnimationTime(0) and then directly access the vertex positions of the model at this time. This will usually not be offered as an API, though - mainly because the vertex positions are usually computed in the shader...

[donmccurdy]

My first reaction here was to apply a weighted blend of the IBMs to each vertex's position, normal, and tangent attributes (with the relevant joint weights), essentially baking the IBMs into the vertex data. This is simple enough.¹ But removing the IBMs this way would be incorrect (whether subtly or dramatically, I'm not sure ...) when a vertex is influenced by different joints with different IBMs.

I imagine the correct way, then, would be to apply the IBMs to the joint node local transforms, and to any animation channels that act on the joint nodes, without modifying the vertex data at all. Implementation details aside, does that sound correct? It's not obvious to me yet whether this option is lossless.

Footnotes

1. Aside, with KHR_mesh_quantization we do the opposite for skinned meshes, baking the vertex dequantization matrix into the IBMs. ↩

[scurest]

My first reaction here was to apply a weighted blend of the IBMs to each vertex's position, normal, and tangent attributes (with the relevant joint weights), essentially baking the IBMs into the vertex data.

This is what the Blender importer does to retarget a skinned mesh onto a new bind pose. Actually it does the complete skinning formula in the new bind pose. It is correct that it is lossy in general but is lossless when the vertex has a single influence. It's also lossless when the new IBMs are related to the old IBM by a constant transform (like an axis conversion or a scale), which is why the importer tries to guess a rest pose that is as close to the IBMs as possible.

Specifically... Write $M_p(n)$ for the world matrix at node n in pose p, IBM(n) for the inverse bind matrix at node n (a certain skin being understood fixed), v for the position of a vertex, rest for the desired new bind pose. If v is influenced by a single node n, its skinned world position is $M_p(n) \; \mathrm{IBM}(n) \; v$ and we change the skin and mesh via

$$\begin{align} &M_p(n) \; \mathrm{IBM}(n) \; v \\\ = \; & M_p(n) \; \big(M_\mathrm{rest}(n)^{-1} \; M_\mathrm{rest}(n) \big) \; \mathrm{IBM}(n) \; v \\\ = \, & M_p(n) \; \underbrace{M_\mathrm{rest}(n)^{-1}}_{\mathrm{IBM}^\prime(n)} \; \underbrace{M_\mathrm{rest}(n) \; \mathrm{IBM}(n) \; v}_{v^\prime} \\\ = \, & M_p(n) \; \mathrm{IBM}^\prime(n) \; v^\prime \end{align}$$

I imagine the correct way, then, would be to apply the IBMs to the joint node local transforms

I don't know how this would work.

[UltraEngine]

My first reaction here was to apply a weighted blend of the IBMs to each vertex's position, normal, and tangent attributes (with the relevant joint weights), essentially baking the IBMs into the vertex data. This is simple enough.1 But removing the IBMs this way would be incorrect (whether subtly or dramatically, I'm not sure ...) when a vertex is influenced by different joints with different IBMs.

That is what I was trying to do at one point. I did not see any problems with the approach, but I was unable to turn it into a complete working solution.

for (int v = 0; v < mesh->vertices.size(); ++v)
{
	position.x = 0; position.y = 0; position.z = 0;
	normal.x = 0; normal.y = 0; normal.z = 0;
	for (int k = 0; k < 4; ++k)
	{
		if (mesh->m_vertices[v].boneweights[k] == 0) continue;
		int boneid = mesh->m_vertices[v].boneindices[k];
		wt[k] = float(mesh->m_vertices[v].boneweights[k]) / 255.0f;
		auto& bone = root->skeleton->bones[boneid];
		if (bone == nullptr)
		{
			Print("Bone " + String(boneid) + " is missing.");
			continue;
		}
		vp = mesh->m_vertices[v].position;
		vn = mesh->m_vertices[v].normal;
		auto it = file.gltfinversebindmatrices.find(bone);
		if (it != file.gltfinversebindmatrices.end())
		{
			bonematrix = file.gltfinversebindmatrices[bone].Inverse();
			vp = TransformPoint(vp, identity, bonematrix);
			vp = TransformPoint(vp, bone->mat, file.rootmatrix.Inverse());
			vn = TransformNormal(vn, identity, bonematrix);
			vn = TransformNormal(vn, bone->mat, file.rootmatrix.Inverse());
		}
		position += vp * wt[k];
		normal += vn * wt[k];
	}
	mesh->m_vertices[v].position = position;
	mesh->m_vertices[v].normal = normal.Normalize();
}

Another approach might be to just make the bone default orientations the inverse of the inverse bind matrices. It will look weird until you apply animation, but that would be fine. Maybe keep the original bone orientations as an extra one-frame animation.

[donmccurdy]

I suspect I've got something wrong here; my results look like the usual skinning horrors (attached), but here's how I assumed this might be implemented:

https://gist.github.com/donmccurdy/0191171b585d797b19a16f1f7633e7b2

Screenshots

The models are "CesiumMan" and "Fox" from https://github.com/KhronosGroup/glTF-Sample-Assets.

It differs a little from @UltraEngine's solution, as I only applied the joint IBM matrix to the vertex. Perhaps I've misunderstood the spec language below?

[IBMs are] used to bring coordinates being skinned into the same space as each joint.

[UltraEngine]

I loaded those two models in Unwrap3D and resaved them. Unwrap3D still includes the inverse bind matrices, but it uses the initial pose as the bind pose, so the IBM is just the inverse of the default pose. I replaced the "inverseBindMatrices" keys in the JSON file with "inverseBindMatrices_" and the models both load correctly in my loader.

https://www.ultraengine.com/files/GLTF_IBM.zip

Maybe that will help!

Conceptually, I think you are transforming the vertex position from the mesh's node space to the binding pose (the inverse of the inverse bind matrix), then transforming that position from the default orientation of the bone (not the binding pose) back to the node's space. Multiply each position by the bone weight, making sure the sum of the weights is 1.0, add the results together, and that's your vertex position.