Kinect mocap video and issues: hands orientation, feet vs SpineBase, fingers

[j2l]

Hi,

Here's my first video of MakeHuman 1.2 Kinect function: https://youtu.be/Z62FqZwZfJo
After the wow effect (I was wowed!), you can see that:

• movement of the SpineBase is not existent, it stays static, we human tend to bend 😸
• therefore feet are not staying on the ground despite being on the ground when motion was captured
• fingers have an orientation issue by default after converting default rig
  

Beside all that, it seems great!

One final note:
Record and stop button should be a toggle button. It's easier to leave your mouse pointer on the button, put yourself in place for mocap with mouse in hand, click to start recording, click again to stop (you can't move the mouse, only click). That's the indie poor lonesome way.

[joepal1976]

Thanks for the report. I'm going to assign this issue to the guy who actually has a kinect to test with. But with the holidays, chances are it might take a while to get a response.

[j2l]

No problemo, thanks :)

[Palmer-JC]

Yes, I know about this. There are some other things too. One being that feet on ground has been redefined for human created via socket. If only one is to be supported, this one will be it. MHX2, FBX, collada, sorry.

[j2l]

Happy new year folks!
Sorry @Palmer-JC, I don't get it:

One being that feet on ground has been redefined for human created via socket. If only one is to be supported, this one will be it

[joepal1976]

In earlier approaches, the routine for deciding where the ground is (in relation to the character) was implemented inside MakeHuman, with the results then exported to (for example) MHX2.

In the new addon, this calculation has been moved to blender, and is different than the one in MH.

We're still stomping out some of the bugs this may have caused. I'm guessing that the phenomenon with feet not staying on the ground might be related to this.

As a side-note for further reference: The inside-blender-feet-on-ground calculation uses the median point of the "joint-ground" joint cube (ie vertex group) as a reference for the ground plane, and the rig object is then moved +Z to place this point at Z=0.

[j2l]

Oh, ok, it makes more sense for me, thank you!

[j2l]

I hoped alpha 3 had some news on this side.
Do you know when kinect will work?

[Palmer-JC]

I am going to start working this again in early April. Initially, I stopped because my kinect was being put out in the garage. That combine cord is really long. It is the only place I have room. Should have known the sensor hates the cold.

When I resume, going to work soley on Blender 2.80, the new skeleton & feet issue first.

[j2l]

Thanks Jeff!

…

On Thu, Mar 21, 2019 at 9:26 PM Jeff Palmer ***@***.***> wrote: I am going to start working this again in early April. Initially, I stopped because my kinect was being put out in the garage. That combine cord is really long. It is the only place I have room. Should have known the sensor hates the cold. When I resume, going to work soley on Blender 2.80, the new skeleton & feet issue first. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#39 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAD_LdoUcWlp8rNcDEdnEnDvHBkBF0_6ks5vY-rRgaJpZM4Zf75m> .

[AivanF]

Hi, @Palmer-JC ! You told that you'll start fixing this problem in April, are there any results already? The problem is really important!

[Mpcreate]

Hello!

Finally Blender 2.80 has been released. You told that your team will fix Kinect 2.0, but it hasn't happened. I think your addon is great, however I really need properly-working Kinect 2.0. Could you please tell me when I can expect a fixed version of Kinect 2.0?

Thank you in advance!

[Palmer-JC]

Short answer is next week possibly earlier. Longer, these issues were just the tip of the iceberg. What was published, actually manipulated the skeleton to match the sensor. This meant that with temperature / clothes changes, subsequent captures would not work right.

Took a while, but the skeleton never changes. Ended up with Blender doing most of the work, & ditch 2 entire files. I think now, that not needing a special skeleton might even be doable. The next publish will have it still, though.

The DLL now "rings" when the number of bodies being recorded changes. The first body which makes the T-Pose & starts the recording, also sets the x-y origin of the capture where it is standing. Additional bodies, which "T-Pose in" are recorded with the origin of the first. This should make multiple synchronized captures work. This is done, but not published yet in its repo.

Fingers data is really poor for this device. I have just put rotation-limit constraints on during assignment to keep really bad results from happening.

I still need to fine tune root bone location to improve feet from violating the floor, or appear to float, but it is already better than before. These are related to armature differences with the model, primarily height.

Am also not yet happy with how smoothing is working. I redid it write in the assignment process to a character, but still twitchy. When these are done, will put out a new version .

[Palmer-JC]

Might as well use this issue to give status. Have published updated DLL repo .

As far as the floor is concerned, I need the MESH to not go thru the floor, not just the armature. Now that I am looking at this really close, I need to add on whatever amount of mesh is below the floor onto the root.location.z in the animation.

This means I am going to have to find all the meshes with this armature (I have this code already), then iterate through the vertices of each one to find the lowest Z vertex value. Fortunately, all meshes, including the armature, have the same origin. The lowest should be negative. Need to add that back, I think.

[Mpcreate]

Wow, it really looks like only the tip of the iceberg. Your work is very impressive! I Still hope that you will make a fully working support for Kinect 2.0. I’m waiting for an updated version of repository, and then I will send you my review. At the moment this addon shows all the signs of recording, however after recording a new animation appears unnamed in the list, and it can not be renamed or applied to the current skeleton.

Is it a problem with addon, or maybe I'm doing something wrong?

[Mpcreate]

@Palmer-JC, I see you fix the the title/name problem, and now it appears with name «Body 0», but I still can’t apply this animation to the skeleton (with character or without).

[Palmer-JC]

You are still running an old version. My commit from 2 days ago did not update the .zip file. I had changes in 40 files, and just need to commit. I have refactored to both handle all skeleton types (default still has problems), as well as handle different scanners in future, like Kinect Azure, if a dll is written to produce the joint data in the same JSON format. Kinect Azure is even supposed to run on Ubuntu.

From 2 replies before, no frames must have been recorded, or there would have been bodies. Remember to do a T Pose & listen for the chime. There is now an error message when that happens.

Why assign is not enabled is that a skeleton is the not the active object. You do not even need a skeleton in the scene to record. The skeleton that does goes with a body need to be selected, then click assign.

The new interface looks like

Hopefully, soon.

I continued on, and now am getting pretty close. Another change is minor jitter is now fixed when

[Palmer-JC]

Ok, your skeleton looks like CMU. That is why assign not enabled. You need to convert with that version. Do not bother, that code is ancient.

[Palmer-JC]

More progress today. First, I moved much of the checking if objects are acceptable for an operation out of where the associated button is enabled or not. This is not helping anyone. They just know that "something is wrong". For all the mocap operations, they now only require that an armature be made the active object. A specific error msg is displayed to indicate the problem when they click it.

The code for assignment is:

        armature = context.object
        problemMsg = None
        rigInfo = RigInfo.determineRig(armature)
        if rigInfo is None:
            problemMsg = 'Unknown rigs are not supported.'
        elif not rigInfo.isMocapCapable():
            problemMsg = 'Rig is not capable of motion capture.'
        elif rigInfo.hasIKRigs():
            problemMsg = 'Cannot be done while rig has an IK snap-on.'
        elif len(context.scene.MhSensorAnimations) == 0:
            problemMsg = 'No current capture being buffered.'
        elif rigInfo.name == 'Default Rig' and not rigInfo.hasRestTpose():
            problemMsg = 'The default rig can only be assigned when it has a rest T-Pose.'

        if problemMsg is not None:
            self.report({'ERROR'}, problemMsg)
        else:
            do it

Also, I have gotten all the rigs to work, though the default rig must have a T rest pose, validated above. You can now just move an armature up when the shoe mesh is below ground. Nothing need be changed to any mesh.

In general, things look pretty good. Multi-bodies needs some testing, and now I am finding that there is a problem doing multiple actions (see) but think things are wrapping up.

[Mpcreate]

It is great news! I am waiting for the new version of your addon. I’m not sure why, but even I did everything you said, current version (release and source) still doesn’t work for me.

[j2l]

Sounds great!
Thanks!
Will test that as soon as I come back home.

[Palmer-JC]

Well, I pushed a version up with a .zip file in it. It is as good as I can get it. I do not think the results are going to be good enough for my purposes. Am closing this issue. Have to do something else.

[Mpcreate]

This version works! Thank you very much!