27‐03‐2024

Today:

• Brainstorming
• Installing MLAgents package and dependencies
• Starting with test project

Brainstorming

Today, I started brainstorming some potential machine learning applications I could work on for this course. I decided rather quickly on a concept, so the bouldering idea is the most worked out. Those ideas were:

AI learns to climb/bouldering

Agent is placed in a boulder climbing hall. Has 2 arms and 2 legs. Lose points for every time they go outside the starting zone on the floor (or stay there for too long), and the simulation starts over. Gain points for grabbing a climbing point and not touching the floor.

Parameters:

• Amount of climbing points. Correlates with the size of the climbing area.
• Arm strength vs leg strength vs balance between both.
• Time the agent is allowed to rest?

Proposal sentence:

Implement Deep Q-Networks in Unity on a machine learning agent to enable it to learn ‘bouldering’, on walls of increasing difficulty with different parameters for samples (attempts to reach the top), and compare the times the top of the walls has been reached for each difficulty with the total amount of samples for that difficulty, with different parameter values.

AI learns to avoid bullets – Bullet Hell

Get points for every second not getting hit. Lose points for every bullet that did hit. Simulation ends when health <= 0. Should eventually learn to find cover. Parameters:

• Move speed
• Amount of cover points

AI plays The Floor is Lava

Proposal sentence:

Parameters:

• Amount of coins
• Lava appearance speed
• Move speed
• Jump height

Implement Deep Q-Network (?) on a “the floor is lava game” in Unity and let it survive for as long as possible and getting as many coins as possible with different parameters for samples (game simulations played) and move speed vs jump height, coin worth, lava area size, and compare the different results with their respective parameters.

Installing MLAgents package and dependencies

The setup and test project is based on a tutorial from

CodeMonkey

I spent a few hours trying to get the MLAgents package from Unity working, which threw some errors at first. It ended up packages being incompatible with eachother so when that was fixed, I could get started on starting a test project, since this is my first encounter with Machine Learning.

Starting with test project

To familiarize myself with the machine learning concept, I started with a simple "Move to position" AI brain. I wanted to get the basics down first before moving on to my actual concept, which is climbing walls with increasing difficulty. For now, I used the tutorial video mentioned earlier.

public class TestAgent : Agent
{
     [SerializeField] private Transform _targetTransform;

     public override void OnEpisodeBegin()
     {
          transform.position = Vector3.zero;
     }
     
     public override void CollectObservations(VectorSensor sensor)
     {
          sensor.AddObservation(transform.position);
          sensor.AddObservation(_targetTransform.position);
     }
     
     public override void OnActionReceived(ActionBuffers actions)
     {
          float moveX = actions.ContinuousActions[0];
          float moveZ = actions.ContinuousActions[1];

          float moveSpeed = 1f;
          transform.position += new Vector3(moveX, 0, moveZ) * Time.deltaTime * moveSpeed;
     }

     private void OnTriggerEnter(Collider other)
     {
          float reward = 0f;
          if (other.CompareTag("Wall"))
               reward = -1f;

          if (other.CompareTag("Target"))
               reward = 1f;
          
          SetReward(reward);
          EndEpisode();
     }
}

To clarify what an agent in this context means:

An agent is an actor that can observe its environment, decide on the
best course of action using those observations, and execute those actions 
within the environment.

Definition taken from the Agent.cs class in the MLagents package.

To start searching for the target, 2 vectors (agent position and target position) are being passed in CollectObservations(). In reality, these vectors consists of in total 6 floats which are then passed to the brain.

With this code, the agent tries to get positive rewards by colliding with the green cylinder. Colliding with the black walls only gets him a negative reward (of -1, these values of rewards are completely arbitrary and depend on each other to enable a balanced training model).

We can speed up training by using more than just 1 agent. For now, I settled with 16 seperate training environments.

Now, it is time to start training!

InitialTesting.mp4

For visualization, I assign a red material to the ground when the agent receives a negative reward, and a green material when it receives a positive reward. As you can see, all the testing environments are red right now, even after (already, because of mass-testing) 31 episodes. I expect that is because the agent does not know what to look for, and only wanders around. In an attempt to tackle this, I start by placing the target much closer than it was before.

When most agents have collided with it a few times, only green floors should start to appear. Let's see if that expectation holds up.

SuccesfullTesting.mp4

In this demonstration, they still struggle a bit in the beginning but around episode 90 they suddenly all start to understand and we only see green floors from then on. Now, let's copy the brain generated from this training and start using it as our training brain. It looks like this in Unity:

However, this brain is trained to move the agent to the position it knows. When we move the target, it's suddenly unable to find it and will get confused. This is why some randomness as an input is required, to make sure it's able to adapt to environment changes.