01‐05‐2024
- Starting with new concept
With my previous machine learning concept, I couldn't even really get past the initial ragdoll setup where it learns to at least stand up straight (I expect because Configurable Joints are a whole new rabbit hole that diving into doesn't help me improve my knowledge on Machine Learning. Because of that, I decided to start something new.
The initial concept is rather simple, and if I see there is enough time to make the idea more sophisticated, I'll do that since I intend this to be a nice portfolio piece.
Implement the MLAgents package in Unity on a Machine Learning agent to enable it to learn to play a minigame: avoiding falling rocks and collecting valuable items, while the level becomes more difficult in terms of (for example) rock count, falling speed, damage & a dynamic environment as different parameter values, and compare the highscores for each unique set of level parameters.
To create an initial prototype for this, I made this very simple testing environment. Eventually I am planning to replace all visuals with actual assets rather than keep using primitive shapes.
The agent (yellow cube) is for now only able to move on the x-axis, so to left and right. The red wired gizmo box is the area where the falling rocks will spawn.
And here, the 'eyes' of the agent are visualized. It's a sensor that shoots raycasts every x degrees in a certain direction for a certain length. The output of that sensor is automatically detected by the agent (as long as UseChildSensors in the Behavior Parameters component is true.
Using the same training setup as last time (spawning training environments in a grid, with a data dashboard visualizing the training results, and coloured materials depending on training results for each individual environment), I'm teaching the agent to avoid the red blocks.
In this training, there are 16 areas training an agent at the same time. During training, Unity speeds up time with 10x/15x, so I added a "current duration" text field to each environment so it gets a bit easier to estimate how the training is doing. With just the "average duration" data in the AI Dashboard, perception of time would get rather warped because it is so sped up.
Then for the rewarding side, every frame the agent has survived, a reward of 0.05 is given (which seems small but it adds up very quickly assuming 60 frames in a second, so that's a reward of 3 for every second). When an agent's StepCount surpassed the MaxStep, a reward of 0.7 is given, so basically for surviving long enough. However, since this is a bit hard to understand (how much time is x amount of steps?), and therefore gives a warped overview of how succesfull the agent actually is, it's something I want to improve on before moving on to the next task.
One way to do this is by using the CurrentDuration variable for each agent, and when it reaches x seconds, give the reward and end the training. This replaces the StepCount and MaxStep variables.
With that training related update and some UI changes, it seems that the results have stabilized. For the first training iteration, I'd say they are pretty good. Now, I can iterate on this brain and perfect its results.
In this video, you can see the AI running on inference (using the trained brain), and mostly succesfull avoiding the blocks. The target duration of 90 seconds is a bit too high, but the average survival time is still 50 seconds. Good enough to move on.
2024-05-01.11-11-21.mp4
Now that the agent is moving around and avoiding the blocks, it's gotten a bit boring to look at. In the evaluation proposal, I said I wanted the agent to 'collect valuable items'. So that's the next thing I'll implement. Soon, the agent will be able to collect coins. However, the environment needs to be a bit wider to make that a challenge. Right now, it's 16 units (meters) wide so I'll triple that to 48 units.
I added sensors on the left and right side of the agent to detect collectibles. To clarify their position and reach, they have been marked with a blue gizmo. The yellow wired box is where the collectibles will spawn.