28‐03‐2024

Recap from where we left off yesterday:

Currently, the brain is trained to move the agent to only one position. When we move the target, it's suddenly unable to find it and will get confused. This is why some randomness as an input is required, to make sure it's able to adapt to environment changes. To achieve this randomness, we place the agent on a different position every time a new episode begins. We do the same for the target. Yesterday, we copied a brain trained on non-random data. Since we're fixing that issue, it has to be trained on a new brain.

Today:

• Adding randomness element to neural network input
• Creating a plan of approach for a "bouldering" AI

     public override void OnEpisodeBegin()
     {
          transform.localPosition = new Vector3(Random.Range(-3f, 1.5f), 1f, Random.Range(-3.5f, 3.5f));
          _targetTransform.localPosition = new Vector3(Random.Range(-3.5f, 3.5f), 1f, Random.Range(-3.75f, 3.75f));
     }

The agent seems to stay in the same place, so to prevent an episode from taking too long and the AI getting stuck, I set the maximum number of steps an agent can take to 5000, as seen in the screenshot below.

One "step" is equal to one frame in the Physics loop, which is 50 frames per second. That means that 5000 max step / 50FPS = 100 seconds, which seems an acceptable amount of time for the agent to get to the target. When the agent fails to reach the target during that time, the floor turns to purple for visualization purposes.

Quoted from the MLAgents Agent.cs class:

Agents in an environment operate in *steps*. At each step, an agent collects observations,
passes them to its decision-making policy, and receives an action vector in response.

The amount of maxSteps is completely arbitrary as well, just like the reward/punishment amount. It fully depends on how long the developer wants to give the AI the chance to find a solution on its own VS starting over and possibly losing a learning opportunity. Of course, the agent can also be taught that taking too long to do something results in a slight punishment as well.

In the following video, you can see mass-testing taking place again. This time, random agent and target position has been added. A counter for succesfull episodes and a success ratio counter has been added as well. Gradually, the agents start doing better and eventually the total success rate is around 87%. That's pretty good!

Demo.mp4

With the following command, we tell it to load a configuration file and use the brain generated in the previous test.

mlagents-learn config/MoveToGoal.yaml --initialize-from=Test9 --run-id=Test10

That gives us the following result:

DemoWithBrain.mp4

Right off the bat, the agents start hitting the target. The success rate stays at a stable 90%, and with more training that number can be increased even more. But I'm not going to do that, since getting an AI to reach a target in this way is not my endgoal.

Creating a plan of approach for a "bouldering" AI

Now that we have trained our first moving machine learning agent, we can move on to the next objective. And that is to get climbing. To stay organized, it's best to create an initial plan of approach and update that along the way, as I become more experienced with machine learning.

This seems like quite a big task, so I'll try dividing it up into smaller, more manageable problems.

To concretely make a definition:

• Create a machine learning agent capable of reaching a certain height on a wall.

While researching this topic, I found a few interesting videos on the topic of AI and bouldering that helped me with inspiration:

Magnus shocked by world’s most advanced climbing AI

AI vs. Stairs (deep reinforcement learning) While not necessarily related to bouldering, this video shows a machine learning agent that's put in a ragdoll, so perhaps it's to some use to me.

Plan of Approach segments

What does the AI look like, and what is it capable of?

• Some way to grab and hold on to the wall

  • Hands & feet - sensor in each
    • Unity ragdolls?
  • Colliders on the wall

• A way to detect where grabbable points are

  • Raycasts

• The AI can exert force from its hands & feet, to move towards a position.

How are rewards/punishments handled?

• A reward when:
  • it manages to reach a new record height
  • it grabs a stone which has a position above the agents height
  • it has no exhaustion (explained below)
• Some sort of "exhaustion" parameter
  • This should encourage trying to gain stability; holding onto the wall with as many body parts as possible (so preferably both hands and feet)
  • Hanging on to a wall with less than 4 body parts will gradually start to fill up the exhaustion meter.
  • When the exhaustion meter reaches 100, the agent gives up and falls down.
  • Exhaustion grows when the agent tries to counter gravity
    • When the wall is slightly inclined, the agent is able to rest and decrease its exhaustion meter. * However, having an exhaustion meter at 0 will result in punishment. This should prevent the agent from being too comfortable in its spot.
    • When the wall is straight up (no angle), there is no difference in exhaustion acceleration.
    • When the wall is slightly declined, the agent has to make use of all his "muscles" to hold on.
• A punishment when a condition is not satisfied
  • When it touches the ground
  • Becoming "exhausted"

What does the testing environment look like?

• A level
  • Increasing difficulty
  • Should be scalable and yet diverse (randomness implementation)
    • Procedural generation?
      • A wall prefab. Contains script that depending on selected difficulty, places x amount of grabbing stones for every x unit on the wall.
      • Grabbing stone prefabs. Are classed per difficulty.

Brainstorming: generating a bouldering wall

It's been a while since I have taken the Procedural Art course, but I'm sure I can remember some of the basics of procedural generation. First of all, let's define the different difficulties the agent will need to complete to gain the title "Boulderman".

1. Beginner
2. Intermediate
3. Strong climber
4. Expert
5. Boulderman

Let's define what each difficulty looks like.

1. Beginner

• Basic wall structure. Short height
• Plenty of grabbing points to hold on to - 100%
• Slight wall incline. This allows for more stability and lets the agent rest, decreasing the exhaustion meter.

2. Intermediate

• Longer wall height.
• A bit less grabbing points - 80%
• No wall incline.

3. Strong climber

• Even longer wall height
• Rotating wall (cylinder shape)
• Even less grabbing points - 60%
• Wall overhangs

4. Expert

• Collectibles on the climbing course: incentive for riskier climbs
• Even less grabbing points - 50%

5. Boulderman

• Time pressure
• Rocks falling down
• Collectibles on the climbing course: incentive for riskier climbs
• Very little grabbing points - 30%

Training a ragdoll agent to walk

I realized that going from a cube agent moving to a target, to a ragdoll agent climbing a wall is too big of a challenge for now. So, I'm going to do the same with the ragdoll agent as I did with the cube agent: teaching it to reach a target. But this time, the agent has limbs. How exciting!

First off, let's figure out how to create a humanoid ragdoll in Unity. Shouldn't be that hard. I found an official Unity resource, so I'll be using that for the most part.