Question: what do "world rotations" mean in the project?

[orenwang]

First of all, amazing project! Thank you for sharing this valuable work.

I am a little confused, though, about "Compute world positions and rotations from a BVH file" part in the README.

Given a simple bvh test.bvh with an all-zero keyframe but non-zero offsets:

HIERARCHY
ROOT Bone
{
	OFFSET 0.000000 0.000000 0.000000
	CHANNELS 6 Xposition Yposition Zposition Xrotation Yrotation Zrotation
	JOINT Bone.001
	{
		OFFSET 0.000000 0.000000 1.000000
		CHANNELS 3 Xrotation Yrotation Zrotation
		End Site
		{
			OFFSET -0.500000 0.000000 0.500000
		}
	}
}
MOTION
Frames: 1
Frame Time: 0.041667
0.000000 0.000000 0.000000 -0.000000 0.000000 -0.000000 -0.000000 0.000000 -0.000000 

Following code

from pymotion.io.bvh import BVH
from pymotion.ops.forward_kinematics_torch import fk
import torch

bvh = BVH()
bvh.load("test.bvh")

local_rotations, local_positions, parents, offsets, end_sites, end_sites_parents = bvh.get_data()
global_positions = local_positions[:, 0, :]  # root joint
pos, rotmats = fk(
    torch.from_numpy(local_rotations),
    torch.from_numpy(global_positions),
    torch.from_numpy(offsets),
    torch.from_numpy(parents),
)

gives rotmats of value

tensor([[[[1., 0., 0.],
          [0., 1., 0.],
          [0., 0., 1.]],

         [[1., 0., 0.],
          [0., 1., 0.],
          [0., 0., 1.]]]])

which indicates "world rotations" are all zero. So if I'm understanding it correctly rotmats is pre-offsets (a.k.a. without applying the rest pose). Is this expected behavior?

[JLPM22]

Thank you for your feedback and for checking out the project! :)

The rotmats output as identity matrices is expected behavior in your described situation. Given that the keyframe values are all zero, it implies that there are no additional rotations applied to the bones.

As I understand for BVH files:

• local_rotations being zero means no rotation from the initial (rest) pose.
• The offsets provided in the BVH hierarchy define the initial pose of the bones (rest pose).
• The rotmats being identity matrices means that the bones remain in their rest pose without any further rotation.

The BVH implementation is based on the following document: https://research.cs.wisc.edu/graphics/Courses/cs-838-1999/Jeff/BVH
Another way to think about it, rotmats are applied first and then translations (pos).

However, I may be missing something, if you could provide the outcome of the expected behavior.

If you have any further questions or need more clarifications, feel free to ask!

[orenwang]

Yeah I should have provided the behavior that I would expect.

The rotmats being identity matrices means that the bones remain in their rest pose without any further rotation

But lets look at the values of pos above:

tensor([[[0., 0., 0.],
         [0., 0., 1.]]])

They are non-zero. Actually result of pos is exactly what I would expect because even bones are in their rest pose, their world positions may not be zero. Same could be said for their rotations, right? From a rest pose, computed pos being "world" but rotmats being rather "local" is where I am confused.

To further make my point, in Blender we can get a world matrix of the second bone by running:

import bpy

arm_ob = bpy.data.objects["test"]
print(arm_ob.pose.bones[1].matrix)

<Matrix 4x4 (0.5000, -0.7071,  0.5000, 0.0000)
            (0.7071,  0.0000, -0.7071,  0.0000)
            (0.5000,  0.7071,  0.5000,  1.0000)
            (0.0000,  0.0000,  0.0000,  1.0000)>

The position part matches the behavior of PyMotion. But the rotation part does not. That's how I understand world rotations too because even bones are in their rest pose without any further rotations, their world rotations may not be zero.

[JLPM22]

Thank you for the detailed explanation! Now I understand the problem better. In my understanding, it is a problem of conventions, which I did not realize before because right now, we only support the BVH file format.

The BVH file does not explicitly declare the initial orientation of each bone, this should be automatically computed. For example, assuming a primary axis having the direction of the parent-child vector, and then assuming some second axis. From this, an initial rotation is computed, which I believe is what Blender is doing under the hood. The problem is that all this assumes a lot, since the base orientation is not explicitly declared in the BVH files.

For now, PyMotion does not assume any of this, and the base orientation is equal to the identity orientation. It all depends on what we consider our initial bone orientation. I understand this may yield counterintuitive results, such as that some bones may have the parent-child vector aligned with, e.g., the Y-world axis, others with the X-world axis, or any vector in general.

I've been reading about it, and I think it may be desirable to align the parent-child direction of all bones with a single world axis, such as the Y-axis. This will help with the interpretation of rotmats and any 3D object alignment. However, this would change the base joint pose representation from a 3D vector (offsets) to a length value and a 3D rotation. But the behaviour would be similar to Blender.

I will write all this down in my roadmap and think of a flexible way to integrate it with the project for all use cases. Hopefully I can work on it soon! :) I will leave this issue open and come back to it once I find a solution.

I leave this here for future reference: https://staffwww.dcs.shef.ac.uk/people/S.Maddock/publications/Motion%20Capture%20File%20Formats%20Explained.pdf

Thank you for pointing out this issue!