Add XR_FB_hand_tracking_capsules extension wrapper and OpenXRFbHandTrackingCapsules node

[devloglogan]

Once again this PR only functions with Godot 4.3 / PR #87546

Creates an extension wrapper for the XR_FB_hand_tracking_capsules extension, which provides info on a bunch of capsule shapes that can be used while hand tracking.

This is implemented via an OpenXRFbHandTrackingCapsules node, which should be the child of an XRController3D. The node instantiates an AnimatableBody3D for each hand tracking capsule at runtime, and updates their transforms every physics frame. I've set the OpenXRFbHandTrackingCapsules node up with duplicate properties of AnimatableBody3D (not all of them, just the ones that I think made sense), and updating those properties at runtime should also update the children accordingly.

hand-tracking-capsules.mp4

[BastiaanOlij]

It's a good start, I've been planning on adding similar logic to XR tools for awhile now, seeing we can do most of this through normal hand tracking, this just adds some sizing info.

One thing I would look into is not creating an AnimatableBody3D child node but instead having a look at the actual implementation and creating the necessary objects directly on the physics engine. You'll have far more control over the implementation.

Also don't make set_sync_to_physics settable, just set it to false and forget it exists. If sync to physics is true, it means physics starts taking control over positioning of the object and overriding your placement.

Next, don't just update the transform, trust me, I've been trying to make hands work properly since I was sent a leap motion years ago, I've run into all the problem cases, I've tried loads of solutions, I'm close but lacked the time to dedicate to it last year.
Updating the transform basically transports the bone, one moment its here, the next its there. Its only working in your video because you have a simple scene and you're moving your hands nice and slowly. the moment you try this in a real setup and start moving your hand at normal speeds, the approach falls apart. One moment your hand is outside an object, the next it's inside, the physics engine panics, and the object flies off at the speed of sound. HURRAY.

edit watching my 5 year old video back I'm reminding I could actually push the cylinders through the table because of this, it totally breaks the physics engine. You'll see 4 become 3 by the end of the video :P

The only way to properly exert forces on objects is to use AnimatableBody3Ds move_and_collide function, where we attempt to move the object from where it currently is, to where I needs to go, figure out what we're colliding with, exert the right forces, rinse and repeat until the bone is there, or until you determine it can't reach where your hand is.
Owh yes, fix one things, 10 new problems popup:

• RigidBody3D has all this logic but it doesn't offer the control you need, and the logic is not accessible in animatable body, because the physics team is building a physics engine for normal games and doesn't cater to the unique requirements of VR (yet)
• move_and_collide only works for movement, not rotation
• you can't do this logic one bone at a time, you really need to take the whole hand into account, look at the position and finger orientation of the hand in the last frame, lerp the whole hand to the current position and finger orientation constantly checking for collisions, Godot does not support this in physics.
• how do you handle a hand getting stuck?
• how do you handle a player teleporting to a new location or a new scene loading, with the hand stuck in an old location not being able to reach the new players location, seeing your node is oblivious to this
• the list goes on

Doing this right, in a way that actually works for a user, requires a fairly complex implementation, with possible enhancements to the physics engine. We can get close, and we should attempt to get close and call it good enough, as the feedback out of this will help us convince the physics team of what XR needs need to be applied to the physics engine.

Lastly have a look at: https://github.com/GodotVR/godot-xr-tools/blob/master/addons/godot-xr-tools/objects/force_body/force_body.gd and https://github.com/GodotVR/godot-xr-tools/blob/master/addons/godot-xr-tools/hands/collision_hand.gd

These form the basis of the implementation that has most of the above things implemented (not all) but only for the palm of the hand. I'm hoping to add finger collisions to this soon but with a number of "this is good enough but far from perfect" concessions due to missing logic.

[dsnopek]

Given all the issues that @BastiaanOlij mentions, and the fact that a number of them may have different solutions depending on how the developer wants the game to work, would it be better to simply expose the capsule positions and sizes and let the developer decide what to do with them? Maybe XR Tools could accept hand capsule information, and that could come either from the developer (or some defaults?), or pulled from this extension if available?

[BastiaanOlij]

Given all the issues that @BastiaanOlij mentions, and the fact that a number of them may have different solutions depending on how the developer wants the game to work, would it be better to simply expose the capsule positions and sizes and let the developer decide what to do with them? Maybe XR Tools could accept hand capsule information, and that could come either from the developer (or some defaults?), or pulled from this extension if available?

The difficulty is not really knowing where this is going, something I totally forgot about and that we still need to expose is that XrHandJointLocationEXT actually has a radius property that gives us the thickness of a bone/finger. That is already enough to obtain information to both generate an animatable hand mesh and create collision capsules. So I'm not sure what Metas extension offers beyond that.

The real problem however is not the capsule data itself. Its the physics interaction and that we're missing functionality in the physics engine to accurately move a the collision data for a hand through space, check for collisions and react to those.

edit DUH, I stand corrected, when I added access to raw data I did actually add it, see OpenXRInterface.get_hand_joint_radius().

[devloglogan]

@BastiaanOlij just looking for a little clarity. If I follow: you feel a simpler implementation of XR_FB_hand_tracking_capsules that exposes the position/radius data to users would be redundant, because the sphere data from the vendor neutral equivalent could already be transformed into similar capsule data on our end?

[m4gr3d]

@devloglogan Can you bump the compatibility_minimum value in plugin.gdextension to 4.3.

[BastiaanOlij]

@BastiaanOlij just looking for a little clarity. If I follow: you feel a simpler implementation of XR_FB_hand_tracking_capsules that exposes the position/radius data to users would be redundant, because the sphere data from the vendor neutral equivalent could already be transformed into similar capsule data on our end?

Once we implement the logic in something like XR Tools yes, but we'd need to verify the data we get from the core extension is indeed usable

[dsnopek]

With PR godotengine/godot#88639, we could conceivably add capsules data to XRHandTracker and have some place to stash the capsule data from this extension.

However, before we do that, it'd probably be best to do some experimentation to see if we can fully derive the capsules from the radii, OR if the capsule data from this extension provides us with something better or more accurate than we can derive.

[devloglogan]

However, before we do that, it'd probably be best to do some experimentation to see if we can fully derive the capsules from the radii, OR if the capsule data from this extension provides us with something better or more accurate than we can derive.

Sure! Happy to look into this and report back soon.

[m4gr3d]

We should have axis constants you can use instead

[m4gr3d]

Can we reuse the XRHandTracker::Hand enum instead?

[m4gr3d]

You can remove this binding since we're not exposing this singleton.

[m4gr3d]

Do we have a destructor to clear those references?

[devloglogan]

It's my understanding that if a node is added to the scene tree (which these would be immediately), it's not necessary to provide cleanup for it. That would fall to the user. @dsnopek I think we talked about this at one point?

[dsnopek]

Ah, yep, we don't generally clean up child nodes, because they'll automatically get cleaned up when their parent node is freed.

[m4gr3d]

You can remove this registration since the api is exposed by the added node class.

[devloglogan]

@dsnopek Finally got around to checking out how well the capsule data can be derived from the already integrated sphere data. I'm just averaging the radius/position between joints to create the capsules. Sphere-derived is left hand, XR_FB_hand_tracking_capsules is the right hand.

capsules-from-spheres.mp4

They're not identical when done this way. If we wanted to tweak it further so that it would be more identical to the right hand, I'm sure we could. But the left hand does feel more accurate to me, and I'd personally feel more inclined to use the left hand as displayed here.

[devloglogan]

I've made a core PR for providing/consuming hand capsule data in XRHandTracker: godotengine/godot#89289.

Also, I realized that I was actually interpreting the hand capsule data incorrectly, and that godot's capsule height doesn't factor in capsule radius at all (as opposed to the data provided by the extension). The size of the hand tracked capsules makes a lot more sense to me with this fix added. :)

[devloglogan]

Closing, as a different implementation has been merged (#116)