Skip to content

Tutorial 4: Car Neural Network

Jump to bottom
Lio edited this page Jan 1, 2025 · 4 revisions

Introduction

Important

This tutorial starts at the workshop-tutorial-3 branch and the complete solution can be found at workshop-tutorial-4 branch.

This tutorial focuses on the implementation of the CarNN class, which contains the main logics for the neural network.

Note

This tutorial will implement/update the following modules:

Preparation

The CarNN class is already partially implemented as we see in the file, particularlly:

  • serialize: turns the neural network's weights into string so that it can be saved to files.
  • deserialize: create the neural network from the given activation function, weights string, and the initial mutation noise to be passed to mutate function which we will implement later.

__init__ Function

But we also need to implement the __init__ function before actually implementing other functions of the neural network.

The prev_fitness and prev_weights members, and the activation function are rather obvious to what they should be initialized to. However, the weights, the layer_sizes, and the hiddens members need a bit more design.

When weights is provided, we can actually find out the layer_sizes from it (by checking its len and shape). On the other hand, if weights is not provided, then the user must provide the layer_sizes so that we can initialize the weights to some specific sizes.

To find out what should the size of weights be, we need to understand how neural network works, but in essense weights are the linkage between each node in the previous layer to each node in the next layer. Due to mathematicaly reasons, the previous layer node will have an additional node than the actual number of hidden node. See below for a pseudocode presenting the use of weights.

for j, prev_node in [prev[0], prev[1], ..., prev[layer_sizes[i - 1]], additional]:
    for k, curr_node in [curr[0], curr[1], ..., curr[layer_sizes[i]]]:
        weights[i][j][k] is the link of prev_node and curr_node

Note

The exact mathematical reason is explained in the Feedforward Neural Network section.

Then after finding layer_sizes, the hiddens are just the layer_sizes except the input and the output layer (i.e. the middle layers).

    def __init__(
        self,
        activation: ActivationFunc,
        weights: Optional[List[np.ndarray]] = None,
        layer_sizes: Optional[List[int]] = None,
    ):
        """
        Initialize the neural network.

        Either `weights` or `layer_sizes` must be provided.

        :param activation: The activation function
        :param weights: The weights of the neural network
        :param layer_sizes: The sizes of the layers
        """
        # Check if either weights or layer_sizes is provided
        if not weights and not layer_sizes:
            raise ValueError("Either weights or layer_sizes must be provided")

        # There is no previous fitness and weights, so they are None
        self.prev_fitness = None
        self.prev_weights = None

        # Initialize the either with the given weight, or using a normal
        # distribution
        #
        # The width should be the previous size + 1 for (x_0, x_1, ..., x_n, b)
        # and the height should be the current layer size
        self.weights = weights or [
            np.random.normal(size=(_____, _____))
            for prev_size, curr_size in zip(layer_sizes, layer_sizes[1:])
        ]

        # Initialize the layer sizes by using the given layer sizes, or
        # calculating from the weights
        self.layer_sizes = layer_sizes or (
            [
                weights[0].shape[0] - 1,
                *(weight.shape[1] for weight in weights),
            ]
        )

        # Initialize the hidden layers
        #
        # Since the initial values are not important, we can just use zeros
        self.hiddens = [
            np.array([0] * size, dtype=np.float32)
            for size in self.layer_sizes[_____:_____]
        ]

        # Assign the activation function
        self.activation = activation
Solution Code 
    def __init__(
        self,
        activation: ActivationFunc,
        weights: Optional[List[np.ndarray]] = None,
        layer_sizes: Optional[List[int]] = None,
    ):
        """
        Initialize the neural network.

        Either `weights` or `layer_sizes` must be provided.

        :param activation: The activation function
        :param weights: The weights of the neural network
        :param layer_sizes: The sizes of the layers
        """
        # Check if either weights or layer_sizes is provided
        if not weights and not layer_sizes:
            raise ValueError("Either weights or layer_sizes must be provided")

        # There is no previous fitness and weights, so they are None
        self.prev_fitness = None
        self.prev_weights = None

        # Initialize the either with the given weight, or using a normal
        # distribution
        #
        # The width should be the previous size + 1 for (x_0, x_1, ..., x_n, b)
        # and the height should be the current layer size
        self.weights = weights or [
            np.random.normal(size=(prev_size + 1, curr_size))
            for prev_size, curr_size in zip(layer_sizes, layer_sizes[1:])
        ]

        # Initialize the layer sizes by using the given layer sizes, or
        # calculating from the weights
        self.layer_sizes = layer_sizes or (
            [
                weights[0].shape[0] - 1,
                *(weight.shape[1] for weight in weights),
            ]
        )

        # Initialize the hidden layers
        #
        # Since the initial values are not important, we can just use zeros
        self.hiddens = [
            np.array([0] * size, dtype=np.float32)
            for size in self.layer_sizes[1:-1]
        ]

        # Assign the activation function
        self.activation = activation

Feedforward Neural Network

Feedforward neural network refers to neural networks which obtains the value of the next layer's nodes based on the previous nodes value multiplied by some weights.

Mathematics

More specifically, to find the value of the node in the current layer ($y$), we need to multiply the value of each node in the previous layer ($x_i$) by their assigned weights ($w_i$), then sum them together.

And in reality, we also add a bias ($b$) to the summed value.

The most simple form of a feedforward neural network is a linear function in the form of:

$$ y = w_0x_0 + b $$

Or commonly written as:

$$ y = mx + c $$

Where the $w$ is written as $m$, the $b$ is written as $c$, and the index $0$ is ignored since there is only one $x$.

Note

The reason why we only use a linear function (as compared to for example a quadratic function) between nodes is mainly because it is the most simple expression of relationship between values, and that more complicated relationships are instead modelled by having multiple layers.

As we multiply more and more hidden layers in between, the relationship between the input layer and the output layer becomes more and more complicated.

But when there are more $x$ values (i.e. the previous layer has multiple nodes), the form will generalize to:

$$ \begin{aligned} y &= w_0x_0 + w_1x_1 + \cdots + w_nx_n + b \\ y &= \sum_{i=0}^n w_ix_i + b \end{aligned} $$

But when there are also multiple $y$ values (i.e. the current layer also has multiple nodes), the form will further generlize to:

$$ \begin{aligned} y_j &= w_{j,0}x_0 + w_{j,1}x_1 + \cdots + w_{j,n}x_n + b_j \\ y_j &= \sum_{i=0}^n w_{j,i}x_i + b_j \end{aligned} $$

We can then make the formula more simple by adding a $1$ to the list of $x$ values, and adding $b_j$ to the list of $w_{j}$ values.

$$ \begin{aligned} x &= [ x_0, x_1, \cdots, x_n, 1 ] \\ w_j &= [ w_{j,0}, w_{j,1}, \cdots, w_{j,n}, b_j ] \\ y_j &= \sum_{i=0}^{n+1} w_{j,i} x_i \end{aligned} $$

In pseudocode, the above formula will be expressed to:

for j in [0, m]
    for i in [0, n + 1]
        y[j] += w[i][j] * x[i]

Now that we know $y$ is a list of values, $x$ is a list of values, and $w$ is a list of lists of values, there is actually a linear algebra operation made for what we need - dot product.

If we represent $y$ as a vector, $x$ as a vector, and $w$ as a matrix, then $y$ is just the dot product of $w$ and $x$.

$$ \begin{aligned} y &= w x \\ \begin{bmatrix} y_0 \\ y_1 \\ \vdots \\ y_{m-1} \\ y_m \end{bmatrix} &= \begin{bmatrix} w_{0,0} & w_{0,1} & \dots & w_{0,n} & w_{0,n+1} \\ w_{1,0} & w_{1,1} & \dots & w_{1,n} & w_{1,n+1} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ w_{m-1,0} & w_{m-1,1} & \dots & w_{m-1,n} & w_{m-1,n+1} \\ w_{m,0} & w_{m,1} & \dots & w_{m,n} & w_{m,n+1} \\ \end{bmatrix} \begin{bmatrix} x_0 \\ x_1 \\ \vdots \\ x_n \\ 1 \end{bmatrix} \\ \begin{bmatrix} y_0 \\ y_1 \\ \vdots \\ y_{m-1} \\ y_m \end{bmatrix} &= \begin{bmatrix} w_{0,0}x_0 + w_{0,1}x_1 + \dots + w_{0,n}x_n + w_{0,n+1} \\ w_{1,0}x_0 + w_{1,1}x_1 + \dots + w_{1,n}x_n + w_{1,n+1} \\ \vdots \\ w_{m-1,0}x_0 + w_{m-1,1}x_1 + \dots + w_{m-1,n}x_n + w_{m-1,n+1} \\ w_{m,0}x_0 + w_{m,1}x_1 + \dots + w_{m,n}x_n + w_{m,n+1} \\ \end{bmatrix} \end{aligned} $$

It seems to be all good now, except one thing - we only have 2 layers ($x$ and $y$). We must also generalize it to have multiple layers.

Let $h_k$ be layer $k$, then $h_{k,i}$ is the value of the node $i$ in layer $k$. Also let $w_k$ be the weights between $h_{k-1}$ and $h_k$, then $w_{k,j,i}$ is the weight between $h_{k,j}$ and $h_{k-1,i}$.

Then the previous formula will be further generlized.

$$ \begin{aligned} h_k &= w_k \text{concat}\left(h_{k-1}, [1]\right) \\ \begin{bmatrix} h_{k,0} \\ h_{k,1} \\ \vdots \\ h_{k,m-1} \\ h_{k,m} \end{bmatrix} &= \begin{bmatrix} w_{k,0,0} & w_{k,0,1} & \dots & w_{k,0,n} & w_{k,0,n+1} \\ w_{k,1,0} & w_{k,1,1} & \dots & w_{k,1,n} & w_{k,1,n+1} \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ w_{k,m-1,0} & w_{k,m-1,1} & \dots & w_{k,m-1,n} & w_{k,m-1,n+1} \\ w_{k,m,0} & w_{k,m,1} & \dots & w_{k,m,n} & w_{k,m,n+1} \\ \end{bmatrix} \begin{bmatrix} h_{k-1,0} \\ h_{k-1,1} \\ \vdots \\ h_{k-1,n} \\ 1 \end{bmatrix} \\ \begin{bmatrix} h_{k,0} \\ h_{k,1} \\ \vdots \\ h_{k,m-1} \\ h_{k,m} \end{bmatrix} &= \begin{bmatrix} w_{k,0,0}h_{k-1,0} + w_{k,0,1}h_{k-1,1} + \dots + w_{k,0,n}h_{k-1,n} + w_{k,0,n+1} \\ w_{k,1,0}h_{k-1,0} + w_{k,1,1}h_{k-1,1} + \dots + w_{k,1,n}h_{k-1,n} + w_{k,1,n+1} \\ \vdots \\ w_{k,m-1,0}h_{k-1,0} + w_{k,m-1,1}h_{k-1,1} + \dots + w_{k,m-1,n}h_{k-1,n} + w_{k,m-1,n+1} \\ w_{k,m,0}h_{k-1,0} + w_{k,m,1}h_{k-1,1} + \dots + w_{k,m,n}h_{k-1,n} + w_{k,m,n+1} \\ \end{bmatrix} \end{aligned} $$

Note

The previous layer is not really the $x$ in our original formula, because remember our original formula's $x$ has a $1$ at the end for multiplying with the bias!

So we also need to $\text{concat}$ a $1$ to $h_{k-1}$ to make the dot product valid.

We are almost there! There is a final step, we need to apply an activation function on the result before really assigning it to the next layer.

$$ h_k = \text{activate}\left(w_k \text{concat}\left(h_{k-1}, [1]\right)\right) $$

Note

While in some scenario, activation functions may not be needed, but the reason why activation functions are often used in between layers is because it provides nonlinearity to the model and clip the values to a certain range, while still preserving differentiability so that the model can still learn.

Intuitively, it is like asking the neural network to think step by step instead of doing everything in one go by keeping the value in a certain range, e.g. "Am I in danger?" -> "Can I outrun the danger?"/"Can I win against the danger?" -> "Should I run or fight?".

Code

After figuring out the math, here is a summary of what we need to do in the CarNN.activate function:

  1. We need to find the first hidden layer's value self.hiddens[0] from the input parameter by doing the computation $h_0 = \text{activate}\left(w_0 \text{concat}(\dots h_{-1}, [1])\right)$, except we don't have $h_{-1}$ so we replace it with the input.
  2. Then we loop over each adjacent layers $h_k$ and $h_{k-1}$ and apply the same computation $h_k = \text{activate}\left(w_k \text{concat}(\dots h_{k-1}, [1])\right)$.
  3. The output layer will be obtained in the same way.
  4. Remember to clip the final output to [-1, 1] because it is what AICar._get_input expects.

Here are some useful numpy functions: numpy.concatenate, numpy.dot, numpy.clip. We will also use the self.activation member, which is a function we got from __init__.

    def activate(
        self, inputs: npt.NDArray[np.float32]
    ) -> npt.NDArray[np.float32]:
        """
        Activate the neural network.

        :param inputs: The input vector
        :return: The output vector
        """
        # Check if the input size is correct
        if len(inputs) != self.layer_sizes[0]:
            raise ValueError(
                f"Expected {self.layer_sizes[0]} inputs, got {len(inputs)}"
            )

        # Calculate the first hidden layer
        self.hiddens[0] = _____(
            np._____(np._____((_____, [1.0])), _____)
        )

        # Calculate the remaining hidden layers in a loop
        for i in range(1, len(self.hiddens)):
            self.hiddens[i] = _____(
                np._____(
                    np._____((_____, [1.0])),
                    _____,
                )
            )

        # Calculate the output layer, and also clip the values to [-1, 1]
        # because it is what is required by the car
        return np._____(
            np._____(
                np._____((_____, [1.0])),
                _____,
            ),
            -1.0,
            1.0,
        )
Solution Code 
    def activate(
        self, inputs: npt.NDArray[np.float32]
    ) -> npt.NDArray[np.float32]:
        """
        Activate the neural network.

        :param inputs: The input vector
        :return: The output vector
        """
        # Check if the input size is correct
        if len(inputs) != self.layer_sizes[0]:
            raise ValueError(
                f"Expected {self.layer_sizes[0]} inputs, got {len(inputs)}"
            )

        # Calculate the first hidden layer
        self.hiddens[0] = self.activation(
            np.dot(np.concatenate((inputs, [1.0])), self.weights[0])
        )

        # Calculate the remaining hidden layers in a loop
        for i in range(1, len(self.hiddens)):
            self.hiddens[i] = self.activation(
                np.dot(
                    np.concatenate((self.hiddens[i - 1], [1.0])),
                    self.weights[i],
                )
            )

        # Calculate the output layer, and also clip the values to [-1, 1]
        # because it is what is required by the car
        return np.clip(
            np.dot(
                np.concatenate((self.hiddens[-1], [1.0])),
                self.weights[-1],
            ),
            -1.0,
            1.0,
        )

Integrating Into AICar

Now that the CarNN.activate function is implemented, we can start integrating it into the AICar class.

As you may have noticed, the class has 2 unused members outputs: npt.NDArray[np.float32] and nn: CarNN. We are going to create them in __init__, and for nn we will create by calling CarNN or CarNN.deserialize based on whether the weights parameter is provided or not.

    def __init__(
        self,
        sensor_rots: npt.NDArray[np.float32],
        activation: ActivationFunc,
        weights: Optional[str] = None,
        init_mutate_noise: float = 0.0,
        hidden_layer_sizes: Optional[List[int]] = None,
        color: pygame.Color = pygame.Color(0, 0, 0),
        out_of_track_color: pygame.Color = pygame.Color(255, 0, 0),
        sensor_color: pygame.Color = pygame.Color(255, 0, 0),
    ):
        # ...
        
        #vvv NEW vvv#
        
        # Check if either weights or hidden_layer_sizes is provided
        if not weights and not hidden_layer_sizes:
            raise ValueError(
                "Either weights or hidden_layer_sizes must be provided"
            )
            
        #^^^ NEW ^^^#

        # Call the base class constructor, i.e. `Car.__init__`
        super().__init__(
            color=color,
            out_of_track_color=out_of_track_color,
        )

        # Initialize the inputs, which is of length `len(sensor_rots) + 2`
        self.inputs = np.array(
            [0.0] * (len(sensor_rots) + 2), dtype=np.float32
        )
        
        #vvv NEW vvv#

        # Initialize the outputs
        self.outputs = vec(0, 0)

        # Initialize the neural network
        #
        # The first layer size must be the length of the inputs, followed by
        # the hidden layer sizes, and finally the length of the outputs
        #
        # And deserialize the weights if provided
        self.nn = (
            CarNN(
                activation=activation,
                layer_sizes=[len(_____), *_____, _____],
            )
            if not weights
            else CarNN.deserialize(
                activation=activation,
                string=weights,
                init_mutate_noise=init_mutate_noise,
            )
        )
            
        #^^^ NEW ^^^#

        self.forward = 0.0
        self.turn = 0.0

        self.sensor_rots = sensor_rots
        self.sensor_color = sensor_color
Solution Code 
    def __init__(
        self,
        sensor_rots: npt.NDArray[np.float32],
        activation: ActivationFunc,
        weights: Optional[str] = None,
        init_mutate_noise: float = 0.0,
        hidden_layer_sizes: Optional[List[int]] = None,
        color: pygame.Color = pygame.Color(0, 0, 0),
        out_of_track_color: pygame.Color = pygame.Color(255, 0, 0),
        sensor_color: pygame.Color = pygame.Color(255, 0, 0),
    ):
        # ...
        
        #vvv NEW vvv#
        
        # Check if either weights or hidden_layer_sizes is provided
        if not weights and not hidden_layer_sizes:
            raise ValueError(
                "Either weights or hidden_layer_sizes must be provided"
            )
            
        #^^^ NEW ^^^#

        # Call the base class constructor, i.e. `Car.__init__`
        super().__init__(
            color=color,
            out_of_track_color=out_of_track_color,
        )

        # Initialize the inputs, which is of length `len(sensor_rots) + 2`
        self.inputs = np.array(
            [0.0] * (len(sensor_rots) + 2), dtype=np.float32
        )
        
        #vvv NEW vvv#

        # Initialize the outputs
        self.outputs = vec(0, 0)

        # Initialize the neural network
        #
        # The first layer size must be the length of the inputs, followed by
        # the hidden layer sizes, and finally the length of the outputs
        #
        # And deserialize the weights if provided
        self.nn = (
            CarNN(
                activation=activation,
                layer_sizes=[len(self.inputs), *hidden_layer_sizes, 2],
            )
            if not weights
            else CarNN.deserialize(
                activation=activation,
                string=weights,
                init_mutate_noise=init_mutate_noise,
            )
        )
            
        #^^^ NEW ^^^#

        self.forward = 0.0
        self.turn = 0.0

        self.sensor_rots = sensor_rots
        self.sensor_color = sensor_color

We also need to import the vec function for initializing the outputs member.

from engine.activations import ActivationFunc
from engine.car_nn import CarNN
from engine.entity.camera import Camera
from engine.entity.car import Car
from engine.entity.track import Track
from engine.utils import dir, vec  # UPDATED

We will then need to call the CarNN.activate function from the AICar._get_input function to control the car.

    def _get_input(self) -> Car.Input:
        # Prepare inputs to the neural network
        #
        # Normalize the speed and angular speed by their maximum values so
        # they are within [-1, 1]
        self.inputs[0] = self.speed / self.MAX_SPEED
        self.inputs[1] = self.angular_speed / self.MAX_ANGULAR_SPEED

        self.outputs = self.nn.activate(_____)

        # Assign outputs of neural network to inputs of the car
        self.forward = self.outputs[0]
        self.turn = self.outputs[1]

        return Car.Input(self.forward, self.turn)
Solution Code 
    def _get_input(self) -> Car.Input:
        # Prepare inputs to the neural network
        #
        # Normalize the speed and angular speed by their maximum values so
        # they are within [-1, 1]
        self.inputs[0] = self.speed / self.MAX_SPEED
        self.inputs[1] = self.angular_speed / self.MAX_ANGULAR_SPEED

        self.outputs = self.nn.activate(self.inputs)

        # Assign outputs of neural network to inputs of the car
        self.forward = self.outputs[0]
        self.turn = self.outputs[1]

        return Car.Input(self.forward, self.turn)

If now you run the program in gameplay mode and load a neural network file, you should see the AI cars drive on along the road. Try python main.py game -t demo0 -n demo -a 5 --follow-ai!

For the record, an argument check is also added to the game.main module.

def main_scene(args: argparse.Namespace):
    """
    Main scene for playing the game

    :param args: The arguments
    :return: None
    """
    
    #vvv NEW vvv#
    
    # Argument check
    #
    # If `args.follow_ai` is True and `args.nn` is not provided, raise a
    # ValueError
    if args.follow_ai and not args.nn:
        raise ValueError(
            "AI follow mode `--follow-ai` requires neural network"
            " `--neural-network`"
        )
        
    #^^^ NEW ^^^#

Evolutionary Algorithm & Gradient Descent

Our reinforcement learning model is going to use evolutionary algorithm, which mimicks how natrual selection works with reproduction and mutation.

However, on top of the evolutionary algorithm, we will make the AI learn faster by also integrating gradient descent, where the neural network approximate the direction of the change needed to be made to the weights to give a better result in the next generation.

Mutation & Gradient Descent

We are going to first implement the CarNN.mutate function.

Before we actually implement the mutation, the function also has another feature - to initialize the neural network by simplying applying some noise to it. By noise, we mean to add or subtract a random value, where the random value is generated by a normal distribution.

When noise is higher, the scale of the normal distribution should be larger, and vice versa when noise is smaller. The numpy.random.normal nicely provide a scale argument for this.

    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
        """
        Mutate the neural network.

        :param noise: The noise
        :param learn_rate: The learning rate
        :param curr_fitness: The current fitness
        :return: None
        """
        
        #vvv NEW vvv#
        
        # If either there is no previous weights, or the current fitness is
        # none, or the learning rate is zero, only noise will be added
        if (
            self.prev_weights is None
            or curr_fitness is None
            or learn_rate == 0
        ):
            # Add noise centering around 0 to the weights
            for i in range(len(self.weights)):
                self.weights[i] += np.random.normal(
                    loc=_____, scale=_____, size=_____
                )
            return
        
        #^^^ NEW ^^^#
Solution Code 
    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
        """
        Mutate the neural network.

        :param noise: The noise
        :param learn_rate: The learning rate
        :param curr_fitness: The current fitness
        :return: None
        """
        
        #vvv NEW vvv#
        
        # If either there is no previous weights, or the current fitness is
        # none, or the learning rate is zero, only noise will be added
        if (
            self.prev_weights is None
            or curr_fitness is None
            or learn_rate == 0
        ):
            # Add noise centering around 0 to the weights
            for i in range(len(self.weights)):
                self.weights[i] += np.random.normal(
                    loc=0, scale=noise, size=self.weights[i].shape
                )
            return
        
        #^^^ NEW ^^^#

For the actual mutation, we will not just randomly mutate our weights between each generation, we will do a gradient descent to approximate how should we mutate.

Our gradient descent method is rather simple and intuitive: if the change between the previous generation and current generation increased the fitness of the AI, then we mutate similar to the previous changes, otherwise we try to mutate in the opposite way.

First, let's find how the fitness has changed, and also what was the previous change in weights.

    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
    	# ...
        
        #vvv NEW vvv#
        
        # Gradient descent
        #
        # Find the change in fitness and weights
        dfitness = _____ - self._____
        dweights = [
            _____ - _____
            for _____, _____ in zip(self._____, self._____)
        ]

        # Update the previous fitness and weights for the next iteration
        #
        # We don't need these values for the calculation later
        self.prev_fitness = curr_fitness
        self.prev_weights = self.weights
        
        #^^^ NEW ^^^#
Solution Code 
    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
    	# ...
        
        #vvv NEW vvv#
        
        # Gradient descent
        #
        # Find the change in fitness and weights
        dfitness = curr_fitness - self.prev_fitness
        dweights = [
            weight - prev_weight
            for weight, prev_weight in zip(self.weights, self.prev_weights)
        ]

        # Update the previous fitness and weights for the next iteration
        #
        # We don't need these values for the calculation later
        self.prev_fitness = curr_fitness
        self.prev_weights = self.weights
        
        #^^^ NEW ^^^#

Then, we are going to modify the weights. We will just multiply the provided learn_rate with the direction of the fitness change and value of weight change and the noise.

    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
    	# ...
        
        #vvv NEW vvv#
        
        # Get the sign of the change in fitness, which indicates whether the
        # previous change was good or bad
        #
        # The sign multiplied by the change in weights gives a general idea of
        # how much the weights should be changed
        #
        # Then we also multiply the learning rate so taht the change does not
        # overshoot
        #
        # Finally, we multiply by some noise to simulate spontaneous mutation
        sign = _____
        for i in range(len(self.weights)):
            self.weights[i] += (
                _____
                * sign
                * _____
                * np.random.normal(loc=_____, scale=noise, size=dweights[i].shape)
            )
            
        #^^^ NEW ^^^#
Solution Code 
    def mutate(
        self,
        noise: float,
        learn_rate: float = 0,
        curr_fitness: Optional[float] = None,
    ):
    	# ...
        
        #vvv NEW vvv#
        
        # Get the sign of the change in fitness, which indicates whether the
        # previous change was good or bad
        #
        # The sign multiplied by the change in weights gives a general idea of
        # how much the weights should be changed
        #
        # Then we also multiply the learning rate so taht the change does not
        # overshoot
        #
        # Finally, we multiply by some noise to simulate spontaneous mutation
        sign = np.sign(dfitness)
        for i in range(len(self.weights)):
            self.weights[i] += (
                learn_rate
                * sign
                * dweights[i]
                * np.random.normal(loc=1, scale=noise, size=dweights[i].shape)
            )
            
        #^^^ NEW ^^^#

Selection

Next, we are going to do a selection. We only want the most fit AI to survive and reproduce. So we are going to copy the neural network of the most fit ai_cars then mutate these new cars in the next_iter function of train.main module.

To make a deep copy of the neural network, we will need copy.deepcopy.

import argparse
import math
import random
from copy import deepcopy  # NEW
from pathlib import Path
from typing import List, Optional, Tuple

We can see that there are 3 command-line arguments we can use: select_count, mutate_noise, and mutate_learn_rate.

    # Define the next iteration function for the AI cars
    def next_iter():
        
        #vvv NEW vvv#
        
        # Sort the AI cars by fitness in descending order so that we can select
        # the top `args.select_count` cars for the next iteration
        ai_cars.sort(key=lambda x: x.fitness, reverse=True)

        # Mutate the neural networks of the cars that are not selected
        #
        # Loop from `args.select_count` to the end of the list
        #
        # Set the neural network of the i-th car to be a deep copy of the
        # `i % args.select_count`-th car's neural network
        #
        # Mutate with `args.mutate_noise`, `args.mutate_learn_rate`, and the
        # car's own fitness
        for i in range(args._____, len(ai_cars)):
            ai_cars[i].nn = deepcopy(ai_cars[_____].nn)
            ai_cars[i].nn.mutate(
                args._____,
                args._____,
                ai_cars[i]._____,
            )
            
        #^^^ NEW ^^^#
        
        # Reset the state of each car
        for car in ai_cars:
            car.reset_state(track)
Solution Code 
    # Define the next iteration function for the AI cars
    def next_iter():
        
        #vvv NEW vvv#
  
        # Sort the AI cars by fitness in descending order so that we can select
        # the top `args.select_count` cars for the next iteration
        ai_cars.sort(key=lambda x: x.fitness, reverse=True)

        # Mutate the neural networks of the cars that are not selected
        #
        # Loop from `args.select_count` to the end of the list
        #
        # Set the neural network of the i-th car to be a deep copy of the
        # `i % args.select_count`-th car's neural network
        #
        # Mutate with `args.mutate_noise`, `args.mutate_learn_rate`, and the
        # car's own fitness
        for i in range(args.select_count, len(ai_cars)):
            ai_cars[i].nn = deepcopy(ai_cars[i % args.select_count].nn)
            ai_cars[i].nn.mutate(
                args.mutate_noise,
                args.mutate_learn_rate,
                ai_cars[i].fitness,
            )
            
        #^^^ NEW ^^^#
        
        # Reset the state of each car
        for car in ai_cars:
            car.reset_state(track)

And that's all! If you run the program in training mode, you should see that after each iteration, the AI behaves slightly different. Try python main.py train -t demo0 -n demo or with a fresh neural network python main.py train -t demo0 demo1 demo2 -n my-nn -s -90 -45 0 45 90 -z 4 4 2.

Clone this wiki locally