[VR] Controller-based hand model mesh has incorrect points

[rickfromwork]

In Edge, there is support for VR controllers to send hand joint data instead of controller data, and so if a playground enables hand tracking it will show a hand in place of the motion controller.

The issue is that the mesh appears distorted, similar to when we had issues before around joint coordinate systems in the past (though this time, just the palm appears distorted). I'm also not sure if this is the expected behavior (replacing controllers with a hand just because the hand tracking feature is enabled).

The playground I used is this one, though any that enable hand tracking should suffice. I'm running Edge version 96.0.1054.57 (Official build) (64-bit), and am using simulation on the Mixed Reality Portal to test in VR.

[sebavan]

The main issue is that it might be an Edge issue if the hands appear properly in other implementation. Could you check that the data are all correct ?

@RaananW will be back after the holidays to have a look.

[RaananW]

We still don't support replacing motion controllers with the hand meshes. This is (i assume) is a collision of the two systems - when the hands feature is enabled, and the ".hand" variable of the xr input is available, it will generate a hand model (and will ignore the actual motion controller). I assume the xr input source has both .gamepad and .hand , and we need to decide which one we are actually using (at the moment, we only support the .gamepad, unless there is an easy way to reliably convert between the two, which i am not sure exists).

TL;dr - if it is a motion controller the hands feature should not generate a hand-mesh for the input source. The default behavior should be - motion controller triumphs hand mesh, unless it is an actual hand. I assume the simplest way to detect this is to check if that doesn't have a .gamepad definition. But this will be a part of the investigation.

[RaananW]

Sorry, I stand corrected. This is an XR evolution :-)

OpenXR on windows provides hand postures to the hand module, based on the current buttons state. Seems very much related to #9089
But as @rickfromwork is saying, the hand meshes look like the are stretched incorrectly. @rickfromwork - do you konw if that's an expected behavior? The OpenXR dev demo is actually showing this behavior as well, showing the hand joints right next to the controller.

I can't seem to find any documentation on this behavior, the hand meshes might have a different source node. Still being investigated

[RaananW]

I believe we won't be using this feature any time soon and we should therefore first prevent it from being enabled.

When pressing the trigger, the hand position changes to this:

When pressing the touchpad, it changes to this:

The idle mode is an open hand:

This actually doesn't correspond with the controller structure itself (only one finger presses the trigger), and feels a bit awkward walking around with open hands if no button is used. Using the thumbstick doesn't move the hand at all. It feels "half baked" as it is.

[RaananW]

Closing this for now, as this is not an issue with babylon directly. Please reopen if we are doing something wrong :-)