New in-house full body IK system

[Frooxius]

We're going to implement our own new full body IK system to address a number of issues. There is more information in the issue here: #612

This discussion is to better structure and coordinate feedback for this feature, as there's a number of different aspects to it.

Before you comment, please read the issue first!

[Frooxius]

Particularly problematic rigs

I'm looking to collect some rigs that are particularly problematic in some way - e.g. they have very unusual proportions.

If you have one and you'd like to contribute it to testing, you can post it here. You can share it in two different ways:

• If you're okay with it being shared publicly (and its licensing allows that), post recrec:/// link to the item/folder here (make sure it's marked public!)
• Send the item/link to me in-game through DM's and post a note here saying that you've sent it.
• If you can, please share the original file too (FBX, GLTF...). This will let us import it fresh and compare and would be great help

The rigs will be kept private and will not be shared outside of team for testing. However we might share videos and pictures of it from testing in devlog!

In both cases please include a picture here, ideally with the skeleton visualization enabled, so we can get a good overview of the rigs, without having to go in-game to see it.

[TheNull]

I have a digitigrade Jerry Avali that I use that has lots of issues tracking properly. Right now, this has the bones renamed in the legs to allow for proper importing into Resonite, but I can provide the original version as well.


I'm willing to purchase and provide a license key for the Jerry and will send that along with the avatar and fbx/gltf files for testing. I've sent a contact request as well so I can DM those to you.

[GunGryphon]

My avatar has a very long "foot" bone and relatively short shins. It has problems with flipping the legs backwards when crouching.

This is my old/current rig, which I've been using the most, the bones are directly driven by the default IK


Crouching is entirely busted because the foot is stationary.

This is the new one that I'm in process of setting up in Resonite. It uses fake armature bones and rotation copying.

In full body the default binding will tie your foot tracker rotation to the entire long foot, which means I have to mind where my feet are facing to keep my entire leg in shape. I would like to have it drive the rotation of the "toe" in my model.

If you would like to experiment with it, I could DM a copy of the armature and body mesh.

[GearBell]

RigBot Test Rigs.zip
Here are custom made rigs for testing this new IK. Included are the Spider, Centaur, MultiArm Digigrade, Digigrade, Multiarm Standard, and Long Limbs. Frooxius and development staff members have permissions to test these rigs, I ask that no one else download these please. I will release them to the public at a later time, but for now these are "closed testing".

[Voxian]

I'm so glad to see a taur rig posted into here! I just wanted to point out that the rig gearbell posted is a cervid / equine design and is very different from a canine / feline. I can't give out the full version of my newest model, but I do have a free version of a Foxtaur still up on gumroad that has a somewhat similar rig. The bones that actually deform the mesh are in a similar setup as I still do on my newer models.

https://voxian.gumroad.com/l/foxtaur

The way I've typically handled these rigs is to assign a full fake human to the hind end and then link them together using a dynamic spine.

Then I handle the legs using two sets of IK, this portion of it should be similar to the same issues people have had with digitgrade legs.

I'm very excited to see taurs on this platform! With a full IK system, we should be able to push the automation on these kinds of avatars pretty far.

[Dandicoot]

Hey all! First of all, thank you for reworking the IK system, really appreciate all the hard work the dev team is putting in <3

I've seen issues when it comes to more muscular models in Resonite. I've compiled a video here detailing some of the differences in how the IK functions for muscular models between VRC and Resonite. I've also supplied Froox with the .blend file for the model so that the dev team can mess around with it and experiment.

https://drive.google.com/file/d/1i4575WwMN1IxAkrSxU_pzrbca4ir75N-/view?usp=sharing

[Frooxius]

Helping to test

I'm sure that we'll get lots of people interested in testing this, so I'm starting this thread ahead of time.

Right now, it's way too early, as no code has been written yet and even after the code starts being written, it will take a bit before it's in state that's usable enough for testing, so please stay patient.

Once the system is ready enough for testing, we'll provide instructions and coordinate from there.

For testers, we are looking in particular for these groups:

• Heavy full body users
• Dancers
• Virtual actors / choreography
• Users with atypical avatars (e.g. with strange proportions)

[jae1911]

I'd be down to test, I use FBT pretty much every single time I jump into VR.

[LeSharc]

I play with 4-5 trackers, and passively dance as I play. I don't play as often as some people, though I was really disappointed upon seeing the current IK system's hip tracking (and the calibrator), and so I would love to test out a new system.
Apologies in advance, as I'm unsure of what 'testing' entails; however, here's my Discord either way: .sharck

[Ruikio]

If there is room. I'd like to be considered for a tester in this when room is available.

[SmolShorty]

I'd like to be considered for a tester. Have 5-6 trackers and might have more coming soon to test it out in Resonite.

[CraftingCreator]

I am using the SlimeVR trackers and am a very active user in vr, getting up, sitting down, moving around, dancing, even jumping (when was the last time you jumped) so think I would be able to stress a more general use case for the IK. As well as enjoy using Digitigrade avatars, of which I have the Rexouium with restructured bones to have the fake plantigrade bones.

[Frooxius]

Full body calibration system

The full body calibration system can only be as good as the IK itself, so at the moment our focus will be on building the IK itself first.

However as part of this, we'll likely make adjustments and improvements to the calibration process too. However until the IK itself is ready, I'd hold off any heavy testing or feedback, because some issues with the calibration are actually issues with FinalIK and might go away.

[TehTurk]

Posting this more as a consideration (So not right now), when and if there is a remaking of the calibrator. I recommend making it as simple as possible. Folks often were quite turned off by how finicky and lack of click and calibrate we were lacking.

[Frooxius]

@TehTurk Yes, the goal is to make it as simple as possible. As explained in the post, the calibrator is only as good as the IK it calibrates for. So improving the IK would make the calibrator give good results out of the box, without need for a lot of fine-tuning.

We'll revisit though once the IK is all up and running and we can see how well it behaves.

[rainbowfl00f]

Would love a way to lock head when calibrating (unless there is a way and I just don't know about it!). That's prolly my biggest issue right now, because the view point tends to shift since my head is never exactly in the same spot when lining myself up

[HamoCorp]

One big improvement to the calibrator could be to use height for calibration. The current calibrator's feet are locked to the ground. So if you have an uneven floor, if you are using playspace moving tools or if your play space has moved like what commonly happens to inside-out headsets drifting and moving slightly up as a result of the playspace not being reset in a while, then the body size doesn't match.

The current in-game option for height seems to be your real height so it might require separate values for real height and full-body height, This height would be based on the headset to the floor and from my experience of other VR games you usually don't set it to your actual height might be higher or lower and might need to change sometimes for better calibration.

[Frooxius]

@HamoCorp I'm not sure how would the two different heights actually feed into the system - it only needs your real world height.

We could potentially make it base it off the position of head and figure where the feet are from that, but it adds more variables, which can lead to worse calibration.

[Frooxius]

Feature/behavior wishlist

What would you like the IK to be able to handle or do? If you had to pick, what is the most important aspect of the IK to make it feel good?

We won't likely be able to address everything, but we'll try to get a good idea and make the system general and robust enough, so we want to get a good sampling of ideas.

Please note, that this is for the IK itself! Features that go with calibration or things that are on top of IK (e.g. say a pose system, stuff to do with specifics of trackers - say getting status of tracker battery and such) do not belong here!

A good way to think about the IK is that it gets a set of points where your head, hands, feet and so on are supposed to be and it calculates the skeleton to match those points. Any points that are missing (e.g. you don't have a foot or hips tracker) are calculated and filled in to look as natural as possible. Pretty much everything else is outside of that.

[amplified1]

I'm not sure I see it explicitly mentioned anywhere so I just want to make sure it is, it would be very nice to have support for mechanical joint constraints and spline chain IK. Right now making inorganic avatars and avatars with prosthetics is a pain because composing joints usually requires having puppet bones and doing a lot of manual math to solve the joint backwards from the original IK solve. Making rigged cables and tubes is also not ideal at the moment because the only option is a dynamic bone with a target, which is less performant and also gives inconsistent results, so a proper simple spline IK module would be useful.

[Frooxius]

@amplified1 I don't think we'd make that part of the fullbody IK itself. That would probably be better suited for a separate and more generalized IK system.

I'll see what happens during the design phase, but it feels like this might end up making the IK too complicated and overscoped for its goals.

[amplified1]

Generic components would be ideal but, at least in the case of something like giving a character bendy spline limbs, it might be beneficial for these systems to influence the core IK solve

[Frooxius]

Like I said, I'll see what happens in the design phase, but it might not fit with the design of the IK system, so it would be split out. The primary goal is to handle most common types of avatars efficiently and well, so if this would compromise that goal, it'll get split out into a separate thing.

[gentlecolts]

I've brought up constraints before in #228, but I agree they'd be a useful concept for the IK as well. Most engines I've seen have a generic concepts of constraints anyways (ex blender, unity), so I don't think such a system should be solely ik-focused

[Frooxius]

Comparison videos / screenshots

If you got any comparison videos and screenshots to show particularly problematic poses and how would you like it to behave, you can put them here!

If you can, ideally have the skeleton enabled as well. Try to make sure that the issue is as clearly demonstrated as possible in the video/screenshot. Cut out any unnecessary parts, such us setup or trying to get to the pose - few seconds of footage showing the specific issue work best.

[shiftyscales]

An issue with the current IK which has becoming increasingly problematic with the introduction of arm-mounted facets, is the bone-roll issue experienced when the hand/wrist is posed in particular positions.

Attempting to look at the underside of your forearm will cause the bone to roll/flip, making it very difficult to use the bottom arm facet anchors.

It also causes the wrist and elbow to twist unnaturally while it flips/rolls over its axis.

2024-05-28.11-12-52.mp4

This is a pretty easy pose/IK issue to replicate in a majority of avatars from my past experience.

[Frooxius]

Related issues

Are there any related issues to the IK on this GitHub? Put them here please!

Note that fingers and posing fingers is a separate system and it does not belong here!

[Frooxius]

Questions & Answers

Do you have questions? We might have answers. Sometimes the answers can be "We don't know yet". But regardless, this is the place to ask them!

[TehTurk]

Actually ohzee's comment makes me think of something that's been asked at large but wasn't sure how it'd be implemented, so bringing it up here for consideration.

Would we be able to blend animations or ik poses similar to how unity's mecanim works? I know that's kind of been a bit of a pain point for some folks who've been wanting to do more advanced avatar stuff, but the undrive/drive aspect or being able to blend values would be kind of the tricky spot there in some of my conversations.

[Frooxius]

@TehTurk Not through the IK itself, that'll be additional feature that will come on top of it at some point.

In order to operate, the IK needs to drive and take control of the bones. There will be mechanism to modify these poses post-calculation, but unlikely with the initial implementation of the IK.

In part this is because we'll need a more robust animation system as well.

[TehTurk]

Ok cool thanks for the response. Wasn't sure how the breakdown would end up being in terms of layering the systems.

[Ardescar]

Will there be a Simple non Avatar IK too? For Gadets, Robots etc? Where you just define the bones into a component to drive some robots or pets or special non standard avatar parts? (could also be in the feature wishlist - but it will be a simple standalone ik beside the Avatar IK for special cases.)

[Frooxius]

@Ardescar There will probably be a few general IK components too that spawn out of this (this is mentioned in the initial post).