Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Added PuppetAvatar.cs Script #52

Merged
merged 2 commits into from
May 22, 2020
Merged

Added PuppetAvatar.cs Script #52

merged 2 commits into from
May 22, 2020

Conversation

anantley
Copy link
Collaborator

Added script to apply joint rotations from Azure Kinect to any humanoid avatar in Unity.

The steps to apply are:

  1. go to the sample scene in the unity sample application
  2. add an avatar and make it a humanoid in the inspector
  3. apply the PuppetAvatar script to the avatar
  4. set the public fields for the azure kinect tracker and the pelvis of the display figure
  5. add an empty Animator Controller to your avatar
  6. press play

Still to do:

  1. enable multi skeleton application of joints
  2. handle scale for different sized avatars

@yijiew yijiew self-requested a review May 22, 2020 01:21
@yijiew yijiew merged commit 6398101 into microsoft:master May 22, 2020
@dotMorten
Copy link

Thanks for this @anantley. Any chance you have a script that also works with the RocketBox Avatars recently released? I tried and all I got was a contortionist and had a hard time figuring out the right values to apply.

RocketBox Avatars repo: https://github.com/microsoft/Microsoft-Rocketbox
Article: https://www.microsoft.com/en-us/research/blog/microsoft-rocketbox-avatar-library-now-available-for-research-and-academic-use/

@anantley
Copy link
Collaborator Author

anantley commented May 26, 2020 via email

@genereddick
Copy link

When I run with a random avatar, it applies the correct movements, but the avatar is rotated. I haven't been able to find where the rotation is coming from.
image

@anantley
Copy link
Collaborator Author

anantley commented May 27, 2020 via email

@genereddick
Copy link

This is the asset structure:

image

Does that mean that the Puppet Avatar script should be on the "root" gameObject in this model?

@dotMorten
Copy link

dotMorten commented May 27, 2020 •
edited

@anantley This might be helpful for the RocketBox Avatars to get each joint based on JointId. My problem was mainly with getting rotation and position right. The sample uses a flat hierarchy whereas Rocketbox is more "intricate":

	private Transform GetJoint(JointId id)
	{
		if(id == JointId.Pelvis)
		{
			return transform.Find("Bip01").Find("Bip01 Pelvis");
		}
		else if(id == JointId.SpineNavel)
		{
			return GetJoint(JointId.Pelvis).Find("Bip01 Spine");
		}
		else if(id == JointId.SpineChest)
		{
			return GetJoint(JointId.SpineNavel).Find("Bip01 Spine1").Find("Bip01 Spine2");
		}
		else if(id == JointId.Head)
		{
			return GetJoint(JointId.Neck).Find("Bip01 Head");
		}
		else if(id == JointId.Neck)
		{
			return GetJoint(JointId.SpineChest).Find("Bip01 Neck");
		}
		else if(id == JointId.ClavicleLeft)
		{
			return GetJoint(JointId.Neck).Find("Bip01 L Clavicle");
		}
		else if(id == JointId.ClavicleRight)
		{
			return GetJoint(JointId.Neck).Find("Bip01 R Clavicle");
		}		
		else if(id == JointId.ShoulderLeft)
		{
			return GetJoint(JointId.ClavicleLeft).Find("Bip01 L UpperArm");
		}
		else if(id == JointId.ShoulderRight)
		{
			return GetJoint(JointId.ClavicleRight).Find("Bip01 R UpperArm");
		}
		else if(id == JointId.ElbowLeft)
		{
			return GetJoint(JointId.ShoulderLeft).Find("Bip01 L Forearm");
		}
		else if(id == JointId.ElbowRight)
		{
			return GetJoint(JointId.ShoulderRight).Find("Bip01 R Forearm");
		}
		else if(id == JointId.WristLeft)
		{
			return GetJoint(JointId.ElbowLeft).Find("Bip01 L Hand");
		}
		else if(id == JointId.WristRight)
		{
			return GetJoint(JointId.ElbowRight).Find("Bip01 R Hand");
		}
		else if(id == JointId.HipLeft)
		{
			return GetJoint(JointId.SpineNavel).Find("Bip01 L Thigh");
		}
		else if(id == JointId.HipRight)
		{
			return GetJoint(JointId.SpineNavel).Find("Bip01 R Thigh");
		}
		else if(id == JointId.KneeLeft)
		{
			return GetJoint(JointId.HipLeft).Find("Bip01 L Calf");
		}
		else if(id == JointId.KneeRight)
		{
			return GetJoint(JointId.HipRight).Find("Bip01 R Calf");
		}		
		else if(id == JointId.AnkleLeft)
		{
			return GetJoint(JointId.KneeLeft).Find("Bip01 L Foot");
		}
		else if(id == JointId.AnkleRight)
		{
			return GetJoint(JointId.KneeRight).Find("Bip01 R Foot");
		}		
		else if(id == JointId.FootLeft)
		{
			return GetJoint(JointId.AnkleLeft).Find("Bip01 L Toe0");
		}
		else if(id == JointId.AnkleRight)
		{
			return GetJoint(JointId.FootRight).Find("Bip01 R Toe0");
		}
		return null;
	}

image

@anantley
Copy link
Collaborator Author

anantley commented May 27, 2020 via email

@genereddick
Copy link

@anantley Here is the model
https://1drv.ms/u/s!Ahk_r7KQn8xsg6wcrPtcDL6MoyCVZw?e=V0J5iy

The first model has a -90 x rotation on the root and a 90 y rotation on the pelvis.

The second model has a -90 x rotation on the Bip01 (root) and a -90 x rotation on the Bio01 Pelvis.

Similar issues on another model.
image

image

@genereddick
Copy link

Tested another rig that was Parent gameObject : Pelvis and that one worked, versus the ones that are Parent gameObject : root : pelvis, which did not.

@anantley
Copy link
Collaborator Author

anantley commented May 27, 2020 via email

@anantley
Copy link
Collaborator Author

@dotMorten
The fix for RocketBox is the following

  1. apply humanoid to the model
  2. press configure
  3. in the Hierarchy window select the Bip01 joint
    image
    and zero out the eulers. This is tricky because in my editor they look like they are zero until you hover over the transform window.
    image
    Then select the Bip01 Pelvis joint and set the eulers to 0,-90,-90
    image
    Then hip apply and you should be set to go.

Let me know if you have any problems with that

@anantley
Copy link
Collaborator Author

@genereddick @dotMorten I made a new version of the puppetmaster.cs script that should handle all the cases with the extra root nodes above the pelvis. It would be great if you could try them out on your problem avatars and let me know what happens.
https://gist.github.com/anantley/b584ddd75a038f36ca8d9c297a79670c

@genereddick
Copy link

@anantley I tested with three different avatars. The one that was working previously and the two with the parent:root:pelvis hierarchy. All three work with the correct rotations. Thanks!

Not a serious issue but there is an short period between when the project starts and the body tracking rotations are first applied where the two (previously non-working) models jump to their incorrect rotations before correcting.

@anantley
Copy link
Collaborator Author

anantley commented May 28, 2020 via email

@dotMorten
Copy link

Rocketbox!
Untitled Project

@dotMorten
Copy link

One thing I did have a problem with was that the character doesn't move around. Ie if I walk a step forward, the character doesn't move a step forward. On the upside, I can moonwalk now :)

@margonzalezfranco
Copy link

Hi @dotMorten did you solve that? I have the translations working too if you are interested

Mar

@dotMorten
Copy link

@margonzalezfrance nope I didn't. Would love to hear how you solved it.

@genereddick
Copy link

@margonzalezfrance I'm also interested

@margonzalezfranco
Copy link

Hi, In the late update after updating all the joins of the avatar you have to translate the pelvis

if (currentbone.name == RootPosition.name)
{
GameObject MoCapBone = GameObject.Find(GetKinectObjectName(currentbone.name));
Vector3 v = MoCapBone.transform.position;
finalJoint.localPosition = new Vector3(OffsetX - v.z, OffsetY - v.x, OffsetZ + v.y);
}

For everyone here, we will be releasing a new animator tool specific for rocketbox in unity
thanks

@ElliMary
Copy link

Thanks for all the working basics, examples and scripts!
For a research work I measured the latency for the Kinect and also in this Unity Code Example and unfortunately I got relatively high results (more than 500 ms). For the 3d viewer application of the kinect itself I already had latency values of over 140 ms.
Do others here also have very high latencies in this Unity Code Example? Or also around 140 ms within the 3D Viewer application?
I'm trying to find out what kind of adjustment screws increase the latency between the kinect 3D viewer application and the Unity bodytracking application.
Can anyone give an insight into how computationally intensive the body tracking calculations within the Unity project are?
Thanks a lot in advance!

@anantley
Copy link
Collaborator Author

Hi @ElliMary can you tell me what your gpu is? That can make a big difference for latency.

@ElliMary
Copy link

Hi @anantley, my GPU is NVIDIA GeForce RTX 2070 SUPER.

@Vandor1
Copy link

Vandor1 commented Apr 28, 2022

@anantley

Still to do:

  1. enable multi skeleton application of joints
  2. handle scale for different sized avatars

Any updates on this? I am currently working on a project where I have multiple skeletons - and I want to add separate avatars on top of them. Currently all my avatars are following the motion of one skeleton.

@asergaz
Copy link

asergaz commented May 11, 2022

Related with SO Thread: https://stackoverflow.com/questions/72040748/azure-kinect-dk-body-tracking-mapping-separate-avatars-on-separate-skeletons

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yijiew yijiew yijiew approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
8 participants
@anantley @dotMorten @genereddick @margonzalezfranco @ElliMary @Vandor1 @asergaz @yijiew