22‐04‐2024

Recap from where we left off last time:

• Created a ragdoll and got it "moving"

Today:

• Rather than forcing the ragdoll to stay upright, train the agent to stay up on its own by using rewards and punishments (reinforcement learning).
  Edit: this turned into researching how Unity approached a MLAgent ragdoll implementation.

Getting the ragdoll to keep standing by applying Machine Learning

To achieve this, the agent first needs to know some information:

• how many limbs that should be observed. This is of relevance for the Vector Observation > Space Size field. As shown in the screenshot below, there are 16 limbs to watch, each with Vector3 Position value (therefore, 16 limbs * 3 vector axis = 48 vector observation array elements)

• the location of each of its limbs. This should be the world position, since using the local position is not possible due to the limb being a child of the main Ragdoll Agent gameobject, rather than the environment container object itself.
• whether a limb is touching the ground
• whether that limb should be touching the ground
  • Depending on the limb type, reward or punishment should be given. For example, I do want the feet to touch the ground (= rewarded), but I don't want the hands and other limbs to touch the ground (= punished). To seperate that behaviour, I created a seperate FeetController script, deriving from BodyLimbContainer.

Found the Unity "Walker" example

After struggling a bit with implementing my ragdoll training, I found that Unity included a "Ragdol walking" agent in their MLAgents package. You can see a recording of it below. I'm obviously not going to copy-paste the code, but it can definitely help with helping me get the agent stay upright.

2024-04-22.15-13-30.mp4

Browsing through the project setup and reading the code, it looks like I was in the right direction with setting rewards/punishments for hitting the ground with certain limbs. Changing the provided CharacterJoint to a ConfigurableJoint was also a good choice.

Inspecting 'Walker' setup

The setup of the ragdoll. Pretty much identical to mine, except for some nice-to-haves like the orientation cube and direction indicator. In the example, the ragdoll can already walk. However, I first want to tackle the problem that the ragdoll can't stay up on its own. That should make it a little easier to implement this.

The ConfigurableJoint component values on a random limb. Some notable fields are the X/Y/Z Motion and the Angular X/Y/Z Motion. which are set to either locked or limited. Other than that, really the only values that seem to be important are the Slerp Drive's values.

Providing observations (Neural Network input data)

This first code snippet provides the input of the neural network, but specifically for a body part.

sensor.AddObservation(bp.groundContact.touchingGround); // Is this bp touching the ground

First off, it's told whether or not the ground is being touched by that specific body part. No context whether that is good or bad, just whether it does.

Then, the object (local) velocities of its rigidbody. I suspect the conversion from world space to object/local space has to do with having multiple agents in the scene and therefore needing to use local coordinates, just like I did with the MoveToGoal test.

//Get velocities in the context of our orientation cube's space
//Note: You can get these velocities in world space as well but it may not train as well.
sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.velocity));
sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.angularVelocity));

Here, the local rotation of every body part except the Hips, handL and handR are being observed.

//Get position relative to hips in the context of our orientation cube's space
sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.position - hips.position));

if (bp.rb.transform != hips && bp.rb.transform != handL && bp.rb.transform != handR)
{
    sensor.AddObservation(bp.rb.transform.localRotation);
    sensor.AddObservation(bp.currentStrength / m_JdController.maxJointForceLimit);
}

Here is the full code snippet as well as the CollectObservations() function where CollectObservationBodyPart() is being called.

/// <summary>
/// Add relevant information on each body part to observations.
/// </summary>
public void CollectObservationBodyPart(BodyPart bp, VectorSensor sensor)
{
    //GROUND CHECK
    sensor.AddObservation(bp.groundContact.touchingGround); // Is this bp touching the ground

    //Get velocities in the context of our orientation cube's space
    //Note: You can get these velocities in world space as well but it may not train as well.
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.velocity));
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.angularVelocity));

    //Get position relative to hips in the context of our orientation cube's space
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(bp.rb.position - hips.position));

    if (bp.rb.transform != hips && bp.rb.transform != handL && bp.rb.transform != handR)
    {
        sensor.AddObservation(bp.rb.transform.localRotation);
        sensor.AddObservation(bp.currentStrength / m_JdController.maxJointForceLimit);
    }
}

/// <summary>
/// Loop over body parts to add them to observation.
/// </summary>
public override void CollectObservations(VectorSensor sensor)
{
    var cubeForward = m_OrientationCube.transform.forward;

    //velocity we want to match
    var velGoal = cubeForward * MTargetWalkingSpeed;
    //ragdoll's avg vel
    var avgVel = GetAvgVelocity();

    //current ragdoll velocity. normalized
    sensor.AddObservation(Vector3.Distance(velGoal, avgVel));
    //avg body vel relative to cube
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(avgVel));
    //vel goal relative to cube
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformDirection(velGoal));

    //rotation deltas
    sensor.AddObservation(Quaternion.FromToRotation(hips.forward, cubeForward));
    sensor.AddObservation(Quaternion.FromToRotation(head.forward, cubeForward));

    //Position of target position relative to cube
    sensor.AddObservation(m_OrientationCube.transform.InverseTransformPoint(target.transform.position));

    foreach (var bodyPart in m_JdController.bodyPartsList)
    {
        CollectObservationBodyPart(bodyPart, sensor);
    }
}

Next, I'll look at how the output of the Neural Network is handled. That's done here:

public override void OnActionReceived(ActionBuffers actionBuffers)
{
    var bpDict = m_JdController.bodyPartsDict;
    var i = -1;

    var continuousActions = actionBuffers.ContinuousActions;
    bpDict[chest].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], continuousActions[++i]);
    bpDict[spine].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], continuousActions[++i]);

    bpDict[thighL].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], 0);
    bpDict[thighR].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], 0);
    bpDict[shinL].SetJointTargetRotation(continuousActions[++i], 0, 0);
    bpDict[shinR].SetJointTargetRotation(continuousActions[++i], 0, 0);
    bpDict[footR].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], continuousActions[++i]);
    bpDict[footL].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], continuousActions[++i]);

    bpDict[armL].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], 0);
    bpDict[armR].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], 0);
    bpDict[forearmL].SetJointTargetRotation(continuousActions[++i], 0, 0);
    bpDict[forearmR].SetJointTargetRotation(continuousActions[++i], 0, 0);
    bpDict[head].SetJointTargetRotation(continuousActions[++i], continuousActions[++i], 0);

    //update joint strength settings
    bpDict[chest].SetJointStrength(continuousActions[++i]);
    bpDict[spine].SetJointStrength(continuousActions[++i]);
    bpDict[head].SetJointStrength(continuousActions[++i]);
    bpDict[thighL].SetJointStrength(continuousActions[++i]);
    bpDict[shinL].SetJointStrength(continuousActions[++i]);
    bpDict[footL].SetJointStrength(continuousActions[++i]);
    bpDict[thighR].SetJointStrength(continuousActions[++i]);
    bpDict[shinR].SetJointStrength(continuousActions[++i]);
    bpDict[footR].SetJointStrength(continuousActions[++i]);
    bpDict[armL].SetJointStrength(continuousActions[++i]);
    bpDict[forearmL].SetJointStrength(continuousActions[++i]);
    bpDict[armR].SetJointStrength(continuousActions[++i]);
    bpDict[forearmR].SetJointStrength(continuousActions[++i]);
}

A dictionary with body parts and their keys (being the transforms of the body part) is retrieved from the controller script. Here, ++i increments the current index. It starts at -1, so the first usage of continuousActions will already use index 0. Every axis is a new continuousAction element. Rather than applying a small force to each body part every frame, it seems a JointTargetRotation and a JointStrength variable is being set. SetJointTargetRotation() accepts 3 elements, so euler angles are being used instead of Quaternions. For the strength, it's only 1 float value, rather than one value for every axis.

Setting rewards/punishments

It looks like rewards and punishments are only done in the GroundContact.cs script. Whenever the limb collider makes contact with another collider (in this case, "Ground"), it marks this limb as touchingGround = true. When penalizeGroundContact is enabled, a negative reward (so a punishment) is given to the agent (note that the limb has a direct reference to the Agent component. And when agentDoneOnGroundContact is true, the agent immediately ends the episode and a new one begins. So when any limb except the feet touches the ground, it receives a punishment and/or ends the episode.

/// <summary>
/// Check for collision with ground, and optionally penalize agent.
/// </summary>
void OnCollisionEnter(Collision col)
{
    if (col.transform.CompareTag(k_Ground))
    {
        touchingGround = true;
        if (penalizeGroundContact)
        {
            agent.SetReward(groundContactPenalty);
        }

        if (agentDoneOnGroundContact)
        {
            agent.EndEpisode();
        }
    }
}

/// <summary>
/// Check for end of ground collision and reset flag appropriately.
/// </summary>
void OnCollisionExit(Collision other)
{
    if (other.transform.CompareTag(k_Ground))
    {
        touchingGround = false;
    }
}