19‐04‐2024
- Implemented ragdolls in Unity.
- Teach the ragdoll AI agent to keep standing (independent balancing)
- Teach the ragdoll AI agent to take a stroll around the block :)
- Refresh my memory on
- the basics of sensors
- agent properties
- IActuator & ActuatorComponents
- Rewards/punishments
Last time, I left off at the ragdoll being forced to stay upright by manipulating the ragdoll parameters. Thinking about that now, I would say it's better try getting the agent do that by itself. That means moving the AI agent is a problem that's a bit too large for now, so I'll tackle moving at a later stage.
Inspecting the Behavior Parameters component on my ragdoll agent, I see there's a field called Use Child Sensors. So that means I can & should attach some type of sensor to at least both feet, hands and probably the head as well.
So now I'll figure out what type of sensor to use for this, since there are 8 of them available and I have no idea what each of them means. I will also refresh my memory on the basics of sensors, vector observations, continuous vs discrete actions and agent properties again, since it's been a good few weeks since I worked on this.
For each sensor type, I'll do a bit of research to have a clear idea on which one is the best option for me.
Sensors that use buffers or tensors for observations. A tensor is "a generalization of vectors and matrices and is easily understood as a multidimensional array".
Uses a Camera object to visually see the environment.
- Grayscale: whether the input data is seen as RGB colours or just a 0-1 grayscale float value. Using colours can result in less training time but requires a heavier GPU since the input data is 3x as big.
- Observation Stacks: how many frames the camera can capture before being fed to the Neural Network. This can be set to a higher number than 1 if you want to teach the AI about something to do with the velocity of an object.
"Sensors that break down the world into a grid of colliders to observe an area at a pre-defined granularity/level of detail."
The sensors that observe Match 3 boards (the game where you have to get 3 of the same figures in a row to gain points). Games such as Candy Crush would use this type of sensor if an AI would be playing it, but it's not relevant to my usecase.
Sensor that's used for vision of the agent. Rather than using a camera, raycasts are being sent out from the agent for a certain distance to check if any colliders intersect with the ray.
Works almost the same as the Camera Sensor, except that the Camera first renders to a Render Texture which the sensor then uses, rather than the Camera itself.
This sensor can be used to observe any Vector-type of input. In my usecase, this sensor can be useful to determine where the hands, feet and head are for example. If the agent knows the location of the head and that it's lower than eye-height, it can learn that the head should stay at that height, contributing to the "learning to walk" process.
A vector3 is fed to the NN as 3 floats. So if, as shown in the screenshot below, I want to "observe" 2 vectors, I will need to set the space size field in Behavior Parameters to 6, 3 float values for each vector. The Stacked Vectors field means it will wait until that amount of data has been collected before being used as input for the agent. Observing a specific axis (for example, only translation on the X-axis means only 1 float is required to get passed).
- Continuous: allows for a range of input/output values in [-1, 1] with the amount of decimal accuracy a float variable can reach. It can be used for transformations & translations.
- Discrete: can be seen as very black and white. Only works with integer values. Discrete actions are used to determine what kind of action an agent should perform (for example, 1 = move left, 2 = move right, 3 = jump)
There are 3 different behavior types for the agent:
- Default
- Used to collect training data and feed it to the Neural Network
- Inference
- Uses the provided Neural Network to handle decision making
- Heuristic
- Used to allow the developer control the agent with their own input
To teach the agent how to perform a certain action, I can reward an agent when it accomplishes something, and punish it when it fails to reach the goal. This is how learning is done in reinforcement learning. By doing this, the agent will over time learn the best way to do a task.
For example, when the goal is to reach a certain position without hitting the obstacles in the smallest amount of time possible, there are several opportunities to get rewards/punishment. Obviously, the agent gets a reward when it reaches the position. When it hits a wall or an obstacle, a negative reward (so a punishment) is given. To stimulate the agent reaching the goal as fast as possible, it can receive a very small punishment for every frame it hasn't reached the goal.
CodeMonkey - How to use Machine Learning AI Vision with Unity ML-Agents!
Unity MLAgents GitHub documentation
CodeMonkey - How to Solve a Match-3 with Machine Learning! (Unity ML-Agents)