Project description, final data, analyzations & conclusion
In this project, machine learning training sessions will be performed, where data will be collected to compare different parameters and visualize what parameters result in the best behaviour for the training environment. At first the idea was to compare negative vs positive reinforcement, but while implementing the prototype, I realized that would not result in valid training data since the scenarios would be entirely different. The negative reinforcement learning would only result in a negative cumulative reward (that the agent tries to minimize as much as possible), so the best result the agent would ever get was 0. While the positive reinforcement scenario would theoretically allow the agent to gain sky-rocketing cumulative reward.
So, The decision was made to only compare negative to negative reinforcement but with slightly different parameters each time, and the same for positive reinforcement. And to enable more interesting data, positive and negative reinforcement will be combined resulting in mixed reinforcement learning.
The parameters that have been adjusted are the reward for finding collectibles, and the punishment for colliding with the falling rocks from the sky.
Implement the MLAgents package in Unity on a Machine Learning agent to enable it to learn to play a minigame: avoiding falling rocks, possibly engage into combat with other agents and collect valuable items, while the level becomes more difficult in terms of (for example) rock count, falling speed, damage & a dynamic environment as different parameter values, and compare the highscores for each unique set of level parameters (positive vs negative reinforcement learning).
Eventually, the agent vs agent combat scenario was scrapped (or at least not continued), since there were already a lot of graphs with data. It's still been explored to a far extent but is not included in the final results.
At first, the idea was to implement a custom machine learning algorithm, but after finding the MLAgents package in Unity and the recommendation by the course teacher to use that package, it seemed like a better option. The MLAgents package uses the PPO algorithm (Proximal Policy Optimization). It's a widely used and well-known machine learning algorithm (for example at OpenAI, which is the company that developed it in the first place), because of how it's relatively easy to use and good performance. To get started with machine learning and to base the prototype off, this tutorial from CodeMonkey was followed.
I have always been intrigued by AI (as many others, introduced to it by NPC behaviour in games). So during some Personal Portfolio courses, many group projects and now the Advanced Tools course, I have been exploring different directions the Artificial Intelligence field can go in. That includes behaviour trees, finite state machines, pathfinding algorithms, flocking behaviour, flowfields and machine learning. It is probably the direction I will do my graduation in and perhaps also my specialty after graduation (eventually, very possible I will first become a general Unity Developer). I chose to use the PPO algorithm, since it is the algorithm used by the tutorial I followed, and one of the most well-known reinforcement learning algorithms there is so it seemed like a good starting point for my first encounter with machine learning.
Sufficient, but hopefully a 'good', since I did my best to make the code reusable.
Good, since there are boxplots displaying the distribution of collectible count over several reward sizes, for example.
Good, since I feel like I understand the concept of machine learning enough for this.
Excellent, since one scientific resource has been used for this prototype, explaining delayed gratification (Chadi & Mousannif, 2023), as well as the future iterations that have been written at the end of the report explaining how the behaviour can be made more complex.
Excellent, since the report is done on this Github Wiki, and my commits show descriptive messages and are organized using branches. (I however did not need to merge anything).
PPO is one of the most popular reinforcement learning algorithms out there (Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization).
Source: (Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization)
This diagram visualizes the process of looping the PPO algorithm. The goal is to find the action which yields the highest reward possible. A collection of these actions mapped to certain states of the environment is called a 'policy'. The agent then uses this policy to determine what specific action resulted in the most desirable state (with the highest cumulative reward), which is then applied to the new behaviour of the agent.
- Fully vs Partially Observable
Partially. There are 2 sensors, a CameraSensor for detecting the falling obstacles but it can only detect a limited area in width, height and depth. And a RayPerceptionSensor3D to detect all colliders surrounding the agent, such as walls, other agents, collectibles and health potions.
- Single vs Multiagent
In training, multiagent to speed the training up. They are behaving competitively, because even though they are not aware of eachother, the goal is still to perform better than the last episode.
In inference (which has not been tested for this prototype), it could be both single AND multi agent. In a real level, there could be several AI agents roaming around, or only 1. Depending on how the behaviour evolves, the agents can choose to work together or chase their own goals.
- Deterministic vs Stochastic
Stochastic, because the positions of the falling rocks are completely random (Brownlee, 2020). The same goes for the position of the collectibles. Since the agent is learning during that process, it affects the machine learning algorithm.
- Episodic vs Sequential
Episodic, since the environment resets every 120 seconds. So looking at the big picture, that means there are no long-term consequences. Collecting coins only affect the (almost) immediate next state of the environment.
- Static vs Dynamic
The environment would be classified as both static (the position of the collectibles only change if the agent picks one of them up), and dynamic, since the falling obstacles are fully random
- Discrete vs Continuous
Continuous. In the BehaviorParameters script in Unity, only ContinuousActions collection is being used, since the agent needs the precision that output provides.
Agent type: Goal-based Agent.
Uses an internal model to decide its action based on specific goals. Goals describe the desirable situation.
Initially, the agent does not know what the goal is. But by following the process, the agent will learn what those goals should be (finding collectibles and avoiding obstacles).
While collecting data for negative reinforcement, it was found that the 'high damage' tests didn't yield any useful results for the amount of episodes (=1200) (or at least not significant) while learning from the environment. The agent would keep colliding with rocks, and the episodes would take the same amount of time at the end of training as at the start. That is because after 2 hits, the episode would end which really limited the learning capacity of the agent. For that reason, it was decided to move forward only with low damage combined with either a high or a low punishment for hitting the rocks. This also decreases the amount of graphs/data visualizations, which is a good thing because there are a lot.
So for negative reinforcement, the following scenarios will be tested:
- Low rock damage, high punishment
- Low rock damage, low punishment
- Low rock damage, high punishment with health potions
- Low rock damage, low punishment with health potions.
As described earlier (in another post), the health potions themselves do not provide any reward but rather a health boost which allows the agent to survive longer. This should teach the agent some form of survival instinct. Specifically, it is related to a form of delayed gratification, since it allows the agent to spend a longer amount of time in the environment to maximize its cumulative reward. In hindsight, it seems that using health potions in an environment that only allows negative reinforcement has no effect, since there is no way to maximize its reward, but rather only minimize its negative cumulative reward.
Comparing the data sets of a low punishment (-0.1) vs a high punishment (-0.9), it seems that the low punishment results in a better learning response. For example in the graph that displays the rock hit count, the trendline for a low punishment decreases at a quicker rate than the high punishment. This suggests that the agent has been taught earlier not to hit the rocks. This analysis is supported by its boxplot, since both the median is lower and the total size of the boxplot is larger, meaning there is more variation in the data.
The boxplot for episode duration also shows a much smaller boxplot size, showing that most (exceptions being the outliers ranging to 40 seconds) episodes fulfilled the episode at 120 seconds.
With these data analysises, it can be concluded that a lower/less harsh punishment results in an agent that is more motivated to learn.
As for positive reinforcements, the data sets that have been collected are from the following scenarios:
- low collectible reward (0.1)
- medium collectible reward (0.5)
- high collectible reward (0.9)
Analyzing the total amount of found collectibles and the cumulative reward, it seems that there is a linear connection between the size of the reward and how motivated the agent is to explore the area to find collectibles. This is confirmed by both the trendlines in the graphs, as well as the boxplots. Reading the boxplot, rewarding the player with a high reward (0.9) results in:
- a median of ~2.7 collected coins, while only rewarding 0.5 results in a median of ~1.1.
In the boxplot for the high reward, there are quite a lot of data outliers, meaning that there were several episode instances in which the agents were able to collect up to 19 collectibles. Compared to the lower rewards, that's exceptional performance since those only resulted in 9 collectibles for 0.1 reward and 13 collectibles for 0.5 respectively.
Now, the goal is to test what happens when both positive and negative reinforcement (with health potions vs without) are combined. This gives the agent a purpose (collecting coins) and a danger source (falling rocks), as well as a way to sustain itself (health potions). This should result in a more complex behaviour, since there will be times it must choose between an immediate reward (the coins), avoiding damage and therefore a punishment (the rocks), or risk damage but a guaranteed health boost. Choosing to go for the health potions could result in a higher cumulative reward in the long-term, since having more health means the agent can go on longer collecting coins.
The tests for this are:- Low rock punishment, low collectible reward
- Low rock punishment, high collectible reward
- High rock punishment, low collectible reward
- High rock punishment, high collectible reward
All sessions will have health potions enabled.
In this graph, the amount of health potions that are collected for each episode are visualized. As mentioned before, the health potions themselves do not provide any reward, only a health boost, so no immediate effect on the cumulative reward. This means that the agent has to learn from the long-term affects before it starts seeing the potion as valuable. It is valuable because the health boost allows the agent to survive longer, which in turn allows it to collect more collectibles, resulting in a higher cumulative reward. This is a variation of 'delayed gratification'. A nice description for this is 'sacrificing your happiness today so that you can be happy tomorrow'.
The graph shows that a LowPunish/HighReward environment results in the most health potions being collected. This is clear by taking a point from around episode ~17, which is 1 potion, and the value at episode 1176, where the value is 76 potions (accurate data retrieved from table). As compared to 'LowPunish/LowReward', the value at episode 17 is 0, whereas at episode 1176, it is only 19 potions.
This is confirmed by the boxplot, where the median is above all of the other 3rd quartiles. This means that half of the values are already higher than 75% of 'LowPunish/LowReward', for example.
Here, we can see how well the agent performs collecting coins while avoiding rocks simultaneously. Unsurprisingly, the low rock punishment combined with high coin reward results in the best collecting behaviour. However, an unexpected result is that the 'low punishment, low reward' parameters are placed 2nd in this tier, so even better than the 'high punishment, high reward' parameters.
And as predicted, the 'high punishment, low reward' results in very little collection behaviour activation.
As for the 'rock hitting' data, it is clear that taking more risk by acting greedily towards collectibles results in more rocks being hit, since most behaviours show a rising trend in rock hit count, with the 'low punishment, high reward' again taking the lead. It seems that the 'high punishment, high reward' and 'low punishment, low reward' result in roughly the same behaviour, which is again a rising trendline.
As established before, the 'high punishment, low reward' parameters demand the agent takes very little action besides trying to avoid rocks as best it can. All parameter scenarios are able to teach the agent to live longer, since they all show a rising trendline in agent survival time. However, there is not one behaviour that is ultimately superior to the others, probably because the max amount of time an episode can take (120s) is just too short to show any significant winner.
Reading this graph (mixed reinforcement without health potions), it is very obvious the performance with health potions regarding the collection of coins is much better compared to mixed reinforcement without health potions. As you can see, with a 'low punishment, high reward', the agent is able to collect ~350 coins after 1200 training episodes, whereas the agent is only able to collect anywhere between ~50 and ~150 coins when health potions are disabled.
Similar to the tests with health potions enabled, these tests with health potions disabled show that only the parameters 'high punishment, low reward' has been able to teach the agent that the priority is to avoid the rocks. The 'high punishment, high reward' also occasionally has some good results where the agent gets hit only once or twice, but clearly less efficient. The other parameters (so the ones that have a low punishment) are not enough to teach the agent to avoid rocks.
Corresponding to the rock hit count graph, this shows that only the parameters 'high punishment, low reward' is efficient in teaching the agent to survive (without health potions). All other paramaters show a stable trendline at around ~50 seconds per episode.
As expected, blue (low punishment, low reward) and grey (high punishment, low reward) allow very little cumulative reward building for the agent. On the contrary, orange (low punishment, high reward) and yellow (high punishment, high reward) are both very capable of maximizing its reward.
From this data, it can concluded that the agent has been taught the concept of 'Delayed Gratification' (Conti, 2024), since there were unavoidably moments during the training where it had to choose between an immediate reward (a coin), or sacrificing that immediate gratification and instead go for long-term survival (a small health boost). The health boost itself gave no reward in terms of adding a value to the cumulative reward, which is how the agent determines if its last action yielded any desirable results. In fact, it is a possibility that attempting to collect the potion also carried a risk to be hit by a rock.
Since the agent receives its own health as an input, it eventually learned that more health == better long term results. A small remark can be made that the agent does not care about its max health; even though sometimes the current health would already be maxed out, health potions would still be collected. To further improve this behaviour, a possible iteration could be to provide a very small punishment (something like 0.0001) when the current health is already maxed out and a health potion was still collected. That would teach the agent that health should not be a priority at that moment, but instead should go for the actual goal: collectibles.
Depending on what we want the agent to achieve, different parameters are optimal. For convenience, 2 different endgoals were created in the table, differentiating whether or not health potions have been used.
| Goal | Without health potions | With health potions |
|---|---|---|
| Maximize survival time | HighPunish/LowReward | LowPunish/HighReward, HighPunish/LowReward |
| Maximize cumulative reward | LowPunish/HighReward, HighPunish/HighReward | LowPunish/HighReward |
Counting how often what parameter type occurs most often:
HighPunish/LowReward: ||
LowPunish/HighReward: |||
HighPunish/HighReward: |
It is clear that the parameter type LowPunish/HighReward offers the overall best performance, so the most balanced behaviour.
Possible new developments could be to:
- Fully implement the combat aspect, with several weapon types and different advantages/disadvantages that come with them. In the context of a game, this trained agent could then be added to a PvE arena style game, where the player battles these trained agents for the highest score.
- It is also possible to add another dimension for agent movement (the Y-axis), so both the player and the agents would be allowed to maneuver even more around the level. This would result in a more dynamic environment, a more complex neural network and more interesting AI behaviour.
- Train the agent to ignore health potions when max health is reached; for example by punishing the agent with a very small amount (0.0001).
Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization
Conti, R. (2024, April 16). Delay of gratification | Psychology, Self-Control & Benefits. Encyclopedia Britannica. https://www.britannica.com/science/delay-of-gratification
Code Monkey. (2020, November 29). How to use Machine Learning AI in Unity! (ML-Agents) [Video]. YouTube. https://www.youtube.com/watch?v=zPFU30tbyKs
Yosider. (n.d.-a). ml-agents-1/docs/Training-PPO.md at master · yosider/ml-agents-1. GitHub. https://github.com/yosider/ml-agents-1/blob/master/docs/Training-PPO.md
Brownlee, J. (2020, July 23). What does stochastic mean in machine learning? MachineLearningMastery.com. https://machinelearningmastery.com/stochastic-in-machine-learning/