Project description, final data, analyzations & conclusion
In this project, I will be performing machine learning sessions, where data will be collected to compare different parameters and visualize what parameters result in the best behaviour for the training environment. At first the idea was to compare negative vs positive reinforcement, but while implementing the prototype, I realized that would not result in valid training data since the scenarios would be entirely different. The negative reinforcement learning would only result in a negative cumulative reward (that the agent tries to minimize as much as possible), so the best result the agent would ever get was 0. While the positive reinforcement scenario would theoretically allow the agent to gain sky-rocketing cumulative reward.
So, I decided to only compare negative to negative reinforcement but with slightly different parameters each time, and the same for positive reinforcement. And to enable more interesting data, positive and negative reinforcement will be combined resulting in mixed reinforcement learning.
The parameters that have been adjusted are the reward for finding collectibles, and the punishment for colliding with the falling rocks from the sky.
Implement the MLAgents package in Unity on a Machine Learning agent to enable it to learn to play a minigame: avoiding falling rocks, possibly engage into combat with other agents and collect valuable items, while the level becomes more difficult in terms of (for example) rock count, falling speed, damage & a dynamic environment as different parameter values, and compare the highscores for each unique set of level parameters (positive vs negative reinforcement learning).
I decided not to continue with the agent vs agent combat scenario, since I already have a lot of graphs. I still explored it to a far extent but did not include it in the final results.
I first wanted to implement my own machine learning algorithm, but after finding the MLAgents package in Unity and the recommendation by my teacher to use that package, I decided to go with that. The MLAgents package uses the PPO algorithm (Proximal Policy Optimization). It's a widely used and well-known machine learning algorithm (for example at OpenAI, which is the company that developed it in the first place), because of how it's relatively easy to use and good performance.
I have always been intrigued by AI (as many others, introduced to it by NPC behaviour in games). So during some Personal Portfolio courses, many group projects and now the Advanced Tools course, I have been exploring different directions the Artificial Intelligence field can go in. That includes behaviour trees, finite state machines, pathfinding algorithms, flocking behaviour, flowfields and machine learning. It is probably the direction I will do my graduation in and perhaps also my specialty after graduation (eventually, very possible I will first become a general Unity Developer). I chose to use the PPO algorithm, since it is the algorithm used by the tutorial I followed, and one of the most well-known reinforcement learning algorithms there is so it seemed like a good starting point for my first encounter with machine learning.
PPO is one of the most popular reinforcement learning algorithms out there (Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization).
Source: (Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization)
This diagram visualizes the process of looping the PPO algorithm. The goal is to find the action which yields the highest reward possible. A collection of these actions mapped to certain states of the environment is called a 'policy'. The agent then uses this policy to determine what specific action resulted in the most desirable state (with the highest cumulative reward), which is then applied to the new behaviour of the agent.
To further develop this prototype, I would:
- Fully implement the combat aspect, with several weapon types and different advantages/disadvantages that come with them. In the context of a game (since I have been wondering how machine learning could be applied to an actual game scenario), this trained agent could then be added to a PvE arena style game, where the player battles these trained agents for the highest score.
- It is also possible to add another dimension for agent movement (the Y-axis), so both the player and the agents would be allowed to maneuver even more around the level. This would result in a more dynamic environment, a more complex neural network and more interesting AI behaviour.
While collecting data for negative reinforcement, I found that the 'high damage' tests didn't yield any useful results for the amount of episodes (=1200) (or at least not significant) while learning from the environment. The agent would keep colliding with rocks, and the episodes would take the same amount of time at the end of training as at the start. That is because after 2 hits, the episode would end which really limited the learning capacity of the agent. For that reason, I decided to move forward only with low damage combined with either a high or a low punishment for hitting the rocks. This also decreases the amount of graphs/data visualizations I have, which is a good thing because there are a lot.
So for negative reinforcement, I will visualize the following scenarios:
- Low rock damage, high punishment
- Low rock damage, low punishment
- Low rock damage, high punishment with health potions
- Low rock damage, low punishment with health potions.
As described earlier (in another post), the health potions themselves do not provide any reward but rather a health boost which allows the agent to survive longer. This should teach the agent some form of survival instinct. Specifically, it is related to a form of delayed gratification, since it allows the agent to spend a longer amount of time in the environment to maximize its cumulative reward. In hindsight, I realize that using health potions in an environment that only allows negative reinforcement has no effect, since there is no way to maximize its reward, but rather only minimize its negative cumulative reward.
Comparing the data sets of a low punishment (-0.1) vs a high punishment (-0.9), it seems that the low punishment results in a better learning response. For example in the graph that displays the rock hit count, the trendline for a low punishment decreases at a quicker rate than the high punishment. This suggests that the agent has been taught earlier not to hit the rocks. This analysis is supported by its boxplot, since both the median is lower and the total size of the boxplot is larger, meaning there is more variation in the data.
The boxplot for episode duration also shows a much smaller boxplot size, showing that most (exceptions being the outliers ranging to 40 seconds) episodes fulfilled the episode at 120 seconds.
With these data analysises, I can conclude that a lower/less harsh punishment results in an agent that is more motivated to learn.
As for positive reinforcements, the data sets I collected are from the following scenarios:
- low collectible reward (0.1)
- medium collectible reward (0.5)
- high collectible reward (0.9)
Analyzing the total amount of found collectibles and the cumulative reward, it seems that there is a linear connection between the size of the reward and how motivated the agent is to explore the area to find collectibles. This is confirmed by both the trendlines in the graphs, as well as the boxplots. Reading the boxplot, rewarding the player with a high reward (0.9) results in:
- a median of ~2.7 collected coins, while only rewarding 0.5 results in a median of ~1.1.
In the boxplot for the high reward, there are quite a lot of data outliers, meaning that there were several episode instances in which the agents were able to collect up to 19 collectibles. Compared to the lower rewards, that's exceptional performance since those only resulted in 9 collectibles for 0.1 reward and 13 collectibles for 0.5 respectively.
#
Now, I want to test what happens when both positive and negative reinforcement (with health potions vs without) are combined. This gives the agent a purpose (collecting coins) and a danger source (falling rocks), as well as a way to sustain itself (health potions). This should result in a more complex behaviour, since there will be times it must choose between an immediate reward (the coins), avoiding damage and therefore a punishment (the rocks), or risk damage but a guaranteed health boost. Choosing to go for the health potions could result in a higher cumulative reward in the long-term, since having more health means the agent can go on longer collecting coins.
The tests I will include for this are:- Low rock punishment, low collectible reward
- Low rock punishment, high collectible reward
- High rock punishment, low collectible reward
- High rock punishment, high collectible reward
All sessions will have health potions enabled.
In this graph, the amount of health potions that are collected for each episode are visualized. As mentioned before, the health potions themselves do not provide any reward, only a health boost, so no immediate effect on the cumulative reward. This means that the agent has to learn from the long-term affects before it starts seeing the potion as valuable. It is valuable because the health boost allows the agent to survive longer, which in turn allows it to collect more collectibles, resulting in a higher cumulative reward. This is a variation of 'delayed gratification'. A nice description I found is 'sacrificing your happiness today so that you can be happy tomorrow'.
Here, we can see how well the agent performs collecting coins while avoiding rocks simultaneously. Unsurprisingly, the low rock punishment combined with high coin reward results in the best collecting behaviour. However, an unexpected result is that the 'low punishment, low reward' parameters are placed 2nd in this tier, so even better than the 'high punishment, high reward' parameters.
And as predicted, the 'high punishment, low reward' results in very little collection behaviour activation.
As for the 'rock hitting' data, it is clear that taking more risk by acting greedily towards collectibles results in more rocks being hit, since most behaviours show a rising trend in rock hit count, with the 'low punishment, high reward' again taking the lead. It seems that the 'high punishment, high reward' and 'low punishment, low reward' result in roughly the same behaviour, which is again a rising trendline.
As established before, the 'high punishment, low reward' parameters demand the agent takes very little action besides trying to avoid rocks as best it can. All parameter scenarios are able to teach the agent to live longer, since they all show a rising trendline in agent survival time. However, there is not one behaviour that is ultimately superior to the others, probably because the max amount of time an episode can take (120s) is just too short to show any significant winner.
Chadi, M., & Mousannif, H. (2023). Understanding Reinforcement Learning Algorithms: The Progress from Basic Q-learning to Proximal Policy. . . ResearchGate. https://www.researchgate.net/publication/369759263_Understanding_Reinforcement_Learning_Algorithms_The_Progress_from_Basic_Q-learning_to_Proximal_Policy_Optimization
Conti, R. (2024, April 16). delay of gratification. Encyclopedia Britannica. https://www.britannica.com/science/delay-of-gratification
Code Monkey. (2020, November 29). How to use Machine Learning AI in Unity! (ML-Agents) [Video]. YouTube. https://www.youtube.com/watch?v=zPFU30tbyKs
Yosider. (n.d.-a). ml-agents-1/docs/Training-PPO.md at master · yosider/ml-agents-1. GitHub. https://github.com/yosider/ml-agents-1/blob/master/docs/Training-PPO.md