-
Notifications
You must be signed in to change notification settings - Fork 0
Tutorial 3: AI Car
Important
This tutorial starts at the workshop-tutorial-2 branch and the complete solution can be found at workshop-tutorial-3 branch.
This tutorial focuses on the implementation of the AICar class, which contains the main logics for the neural network.
Note
This tutorial will implement/update the following modules:
There are some properties and functions to be implemented before we implement the update logics and draw logics.
Property in Python is like getter function in other languages, they are used to get value from the class. They are generally marked with the @property decorator, such as the example below which defines a property diameter that returns a value of self.radius * 2.0.
@property
def diameter(self) -> float:
return self.radius * 2.0And then the programmer can simply call my_obj.diameter to get the value of the property. Note that assignment operator cannot be used on properties, instead we need to define a property setter to enable assignment to it.
@diameter.setter
def diameter(self, value: float):
self.radius = value / 2.0There are some properties in AICar for us to implement so that information can be used later for the neural network.
First, the AICar.fitness returns it's superclass' Car.progress normalized into the range [0, 1].
Solution Code
@property
def fitness(self) -> float:
"""
Get the fitness of the AI car.
Equal to the normalized progress.
:return: The fitness of the AI car
"""
return self.progress / self.total_progress # NEWSecond, the AICar.inputs field is used to store the normalized values of Car.speed, Car.turn, and the sensor distances. Since Car.speed and Car.turn are managed by the superclass, we only need to assign them to inputs when we are going to use it.
As for the sensor distances, we are going to assign them when we are updating the car in AICar.update function. So it is better to make a property to make the code later to be more readable.
Solution Code
@property
def sensors(self) -> npt.NDArray[np.float32]:
"""
Get the sensors of the AI car.
Equivalent to `self.inputs[2:]`.
:return: The sensors of the AI car
"""
return self.inputs[2:] # NEW @sensors.setter
def sensors(self, value: npt.NDArray[np.float32]):
"""
Set the sensors of the AI car.
Equivalent to setting `self.inputs[2:]`.
:param value: The value to set
:return: None
"""
self.inputs[2:] = value # NEWWe also need to initialize the car in the AICar.__init__ we have some more fields than the superclass.
def __init__(
self,
sensor_rots: npt.NDArray[np.float32],
activation: ActivationFunc,
weights: Optional[str] = None,
init_mutate_noise: float = 0.0,
hidden_layer_sizes: Optional[List[int]] = None,
color: pygame.Color = pygame.Color(0, 0, 0),
out_of_track_color: pygame.Color = pygame.Color(255, 0, 0),
sensor_color: pygame.Color = pygame.Color(255, 0, 0),
):
"""
Initialize the AI car.
Either `weights` or `hidden_layer_sizes` must be provided.
:param sensor_rots: The sensor rotations
:param activation: The activation function
:param weights: The weights of the neural network
:param init_mutate_noise: The initial mutation noise
:param hidden_layer_sizes: The hidden layer sizes
:param color: The color of the car
:param out_of_track_color: The border color of the car when it is out
of track
:param sensor_color: The color of the sensor rays
"""
# Call the base class constructor, i.e. `Car.__init__`
super().__init__(
color=color,
out_of_track_color=out_of_track_color,
)
# Initialize the inputs, which is of length `len(sensor_rots) + 2`
self.inputs = np.array(
_____, dtype=np.float32
)
self.forward = 0.0
self.turn = 0.0
self.sensor_rots = sensor_rots
self.sensor_color = sensor_colorSolution Code
def __init__(
self,
sensor_rots: npt.NDArray[np.float32],
activation: ActivationFunc,
weights: Optional[str] = None,
init_mutate_noise: float = 0.0,
hidden_layer_sizes: Optional[List[int]] = None,
color: pygame.Color = pygame.Color(0, 0, 0),
out_of_track_color: pygame.Color = pygame.Color(255, 0, 0),
sensor_color: pygame.Color = pygame.Color(255, 0, 0),
):
"""
Initialize the AI car.
Either `weights` or `hidden_layer_sizes` must be provided.
:param sensor_rots: The sensor rotations
:param activation: The activation function
:param weights: The weights of the neural network
:param init_mutate_noise: The initial mutation noise
:param hidden_layer_sizes: The hidden layer sizes
:param color: The color of the car
:param out_of_track_color: The border color of the car when it is out
of track
:param sensor_color: The color of the sensor rays
"""
# Call the base class constructor, i.e. `Car.__init__`
super().__init__(
color=color,
out_of_track_color=out_of_track_color,
)
# Initialize the inputs, which is of length `len(sensor_rots) + 2`
self.inputs = np.array(
[0.0] * (len(sensor_rots) + 2), dtype=np.float32
)
self.forward = 0.0
self.turn = 0.0
self.sensor_rots = sensor_rots
self.sensor_color = sensor_colorWe also have to do some additional steps in the AICar.reset_state so that some fields can be reset, i.e. the forward, turn, and inputs.
Solution Code
def reset_state(self, track: Track):
"""
Reset the state of the car.
:param track: The track
:return: None
"""
self.forward = 0.0
self.turn = 0.0
self.sensors.fill(self.SENSOR_DIST)To implement the logics for updating and drawing the car, we are going to extend the logics of Car.
In the update logics, the additional logics is the sensor detection. We are going to use the shapely.LinearRing.intersection of Track.shapely_linear_ring to detect where the sensor hits the edge of the track by constructing a shapely.LineString from the car's position to the direction of the AICar.sensor_rots with a length of AICar.SENSOR_DIST.
def update(self, dt: float, track: Track):
"""
Update the AI car.
:param dt: The delta time
:param track: The track
:return: None
"""
# Update sensors
#
# For each global sensor rotation (the sensor rotation plus the car
# rotation), find the intersection of those sensor rays with the edge
# of the track, i.e. `shapely_linear_ring`
#
# Then, for each of these intersections, calculate the distance from
# the car to the intersection, and normalize it by `SENSOR_DIST` so
# they are within [0, 1]
self.sensors = np.array(
[
(
shapely.points(self.pos).distance(intersection)
/ _____
if not intersection.is_empty
else _____
)
for intersection in (
track.shapely_linear_ring.intersection(
shapely.linestrings(
[
self.pos,
self.pos + _____,
]
)
)
for rot in self.sensor_rots + _____
)
],
dtype=np.float32,
)Solution Code
def update(self, dt: float, track: Track):
"""
Update the AI car.
:param dt: The delta time
:param track: The track
:return: None
"""
# Update sensors
#
# For each global sensor rotation (the sensor rotation plus the car
# rotation), find the intersection of those sensor rays with the edge
# of the track, i.e. `shapely_linear_ring`
#
# Then, for each of these intersections, calculate the distance from
# the car to the intersection, and normalize it by `SENSOR_DIST` so
# they are within [0, 1]
self.sensors = np.array(
[
(
shapely.points(self.pos).distance(intersection)
/ self.SENSOR_DIST
if not intersection.is_empty
else 1.0
)
for intersection in (
track.shapely_linear_ring.intersection(
shapely.linestrings(
[
self.pos,
self.pos + (dir(rot) * self.SENSOR_DIST),
]
)
)
for rot in self.sensor_rots + self.rot
)
],
dtype=np.float32,
)In addition, the superclass uses Car._get_input to update the car's movement. So let's also implement the AICar._get_input to simulate calling the neural network.
Solution Code
def _get_input(self) -> Car.Input:
# Prepare inputs to the neural network
#
# Normalize the speed and angular speed by their maximum values so
# they are within [-1, 1]
self.inputs[0] = self.speed / self.MAX_SPEED
self.inputs[1] = self.angular_speed / self.MAX_ANGULAR_SPEED
# TODO: Activate the neural network to
# get the ouotput based on the inputs
# Assign outputs of neural network to inputs of the car
self.forward = 1
self.turn = -1
return Car.Input(self.forward, self.turn)Note
We will implement the actual neural network in tutorial 4, so for now it is just turning left and going forward.
We are now going to implement the draw logics, first we are going to extend the draw logics, and call the AICar.draw_sensor function when the car is not out of track.
Solution Code
def draw(self, screen: pygame.Surface, camera: Camera):
"""
Draw the AI car.
:param screen: The screen to draw on
:param camera: The camera
:return: None
"""
if not self.out_of_track:
self.draw_sensor(screen, camera)
super().update(dt, track)Then, implement the AICar.draw_sensor function by drawing the line according to the AICar.sensors property.
def draw_sensor(self, screen: pygame.Surface, camera: Camera):
"""
Draw the sensor of the AI car.
:param screen: The screen to draw on
:param camera: The camera
:return: None
"""
# Use the sensor rotations and the sensor distances to draw the lines
for rot, dist in zip(
self.sensor_rots + self.rot,
self.sensors * self.SENSOR_DIST,
):
sensor_end = self.pos + _____
pygame.draw.line(
screen,
self.sensor_color,
_____,
_____,
)
super().draw(screen, camera)Solution Code
def draw_sensor(self, screen: pygame.Surface, camera: Camera):
"""
Draw the sensor of the AI car.
:param screen: The screen to draw on
:param camera: The camera
:return: None
"""
# Use the sensor rotations and the sensor distances to draw the lines
for rot, dist in zip(
self.sensor_rots + self.rot,
self.sensors * self.SENSOR_DIST,
):
sensor_end = self.pos + (dir(rot) * dist)
pygame.draw.line(
screen,
self.sensor_color,
camera.get_coord(self.pos),
camera.get_coord(sensor_end),
)
super().draw(screen, camera)Now that the AI car class is fully implemented, let's put them into game.main and train.main.
In gameplay mode, we need to do the same for our AI Car as our player Car in the restart function - reseting its state.
# Define the restart function
def restart():
# If `args.follow_ai` is False, reset the state of the player car by
# calling `reset_state` with the track
if not args.follow_ai:
player_car.reset_state(track)
#vvv NEW vvv#
# Reset the state of each AI car by calling `reset_state` with the
# track
for car in ai_cars:
car.reset_state(track)
#^^^ NEW ^^^#Solution Code
# Define the restart function
def restart():
# If `args.follow_ai` is False, reset the state of the player car by
# calling `reset_state` with the track
if not args.follow_ai:
player_car.reset_state(track)
#vvv NEW vvv#
# Reset the state of each AI car by calling `reset_state` with the
# track
for car in ai_cars:
car.reset_state(track)
#^^^ NEW ^^^#Then, we also need to call the update function, as well as sort the AI cars according to their fitness since the camera will always follow the first car in the ai_cars list.
# Update the player car if `args.follow_ai` is False
if not args.follow_ai:
player_car.update(fixed_dt, track)
#vvv NEW vvv#
# Update each AI car
#
# Skip the car if it is out of track
for car in ai_cars:
if _____:
continue
_____
# If AI follow mode is enabled, sort
# the AI cars by fitness in descending order
if args.follow_ai:
ai_cars.sort(key=lambda x: _____, reverse=_____)
#^^^ NEW ^^^#
# Update the camera to follow the first AI car if `args.follow_ai` is
# True, otherwise follow the player car
camera.update(fixed_dt, args.follow_ai and ai_cars[0])Solution Code
# Update the player car if `args.follow_ai` is False
if not args.follow_ai:
player_car.update(fixed_dt, track)
#vvv NEW vvv#
# Update each AI car
#
# Skip the car if it is out of track
for car in ai_cars:
if car.out_of_track:
continue
car.update(fixed_dt, track)
# If AI follow mode is enabled, sort
# the AI cars by fitness in descending order
if args.follow_ai:
ai_cars.sort(key=lambda x: x.fitness, reverse=True)
#^^^ NEW ^^^#
# Update the camera to follow the first AI car if `args.follow_ai` is
# True, otherwise follow the player car
camera.update(fixed_dt, args.follow_ai and ai_cars[0])Now try it out with python main.py game -t demo0 -n demo -a 5 and you should now see the sensors of the AI cars being drawn, and that they are now going forward and turning left.
In training mode, we are going to do similar things to the gameplay mode.
However in addition, we want to make the training be able to go into next iteration after all the cars are out of track, or when the user press the return key. This way the training can actually go into the next iteration, in other words allowing the AI to retry and learn.
Let's first define a next_iter function that resets the AI cars.
# Reset the state of each car by calling `reset_state` with the track
for car in ai_cars:
car.reset_state(track)
#vvv NEW vvv#
# Define the next iteration function for the AI cars
def next_iter():
# Reset the state of each car
_____:
_____
#^^^ NEW ^^^#
# Main loop forever while `running` is True
running = True
fixed_dt = 0.032
skip_frame_counter = 0
while running:
# ...Solution Code
# Reset the state of each car by calling `reset_state` with the track
for car in ai_cars:
car.reset_state(track)
#vvv NEW vvv#
# Define the next iteration function for the AI cars
def next_iter():
# Reset the state of each car
for car in ai_cars:
car.reset_state(track)
#^^^ NEW ^^^#
# Main loop forever while `running` is True
running = True
fixed_dt = 0.032
skip_frame_counter = 0
while running:
# ...Then, we can call it when the user press the return key, or all cars are out of track. Also, add the same update logics as in the gameplay mode.
# ...
if (
pygame.key.get_mods() & pygame.KMOD_CTRL
and event.key == pygame.K_s
):
# If control + s is pressed, we save the neural network
save_nn(args, ai_cars)
#vvv NEW vvv#
if event.key == pygame.K_RETURN:
# If enter is pressed, we trigger the next iteration
#
# Also randomize the new track before the next iteration
track = random.choice(tracks)
_____
# If all cars are out of track, trigger the next iteration
#
# Also randomize the new track before the next iteration
if _____:
track = random.choice(tracks)
_____
# Update each car
#
# Skip the car if it is out of track
for car in ai_cars:
if car.out_of_track:
continue
car.update(fixed_dt, track)
# Sort the AI cars by fitness in descending order
ai_cars.sort(key=lambda x: x.fitness, reverse=True)
#^^^ NEW ^^^#
# Update the camera to follow the first AI car, i.e. the most fit car
camera.update(fixed_dt, ai_cars[0])
# ...Solution Code
# ...
if (
pygame.key.get_mods() & pygame.KMOD_CTRL
and event.key == pygame.K_s
):
# If control + s is pressed, we save the neural network
save_nn(args, ai_cars)
#vvv NEW vvv#
if event.key == pygame.K_RETURN:
# If enter is pressed, we trigger the next iteration
#
# Also randomize the new track before the next iteration
track = random.choice(tracks)
next_iter()
# If all cars are out of track, trigger the next iteration
#
# Also randomize the new track before the next iteration
if all(car.out_of_track for car in ai_cars):
track = random.choice(tracks)
next_iter()
# Update each car
#
# Skip the car if it is out of track
for car in ai_cars:
if car.out_of_track:
continue
car.update(fixed_dt, track)
# Sort the AI cars by fitness in descending order
ai_cars.sort(key=lambda x: x.fitness, reverse=True)
#^^^ NEW ^^^#
# Update the camera to follow the first AI car, i.e. the most fit car
camera.update(fixed_dt, ai_cars[0])
# ...Now, try it out with python main.py train -t demo0 demo1 demo2 -n my-nn -s -90 -45 0 45 90 -z 4 4 2 -f leaky_relu -q 3 -a 10 -c 3. When all cars are out of track or you press return key, it should now reset automatically.