Skip to content

Tutorial 0: Introduction

Lio edited this page Feb 9, 2025 · 1 revision

Introduction

Welcome to the tutorial of creating a simple reinforcement learning (RL) driver using Python and NumPy. Since we are not going to use any high-level machine learning frameworks, such as TensorFlow, you will be able to understand the fundamentals of reinforcement learning and deep neural networks in general.

Project Scaffold

Although we will be doing most of the programming by ourselves to get the most out of the tutorial, there will be some existing code that are trivial and annoying to implement already implemented for you.

Tutorial 0 Branch

The scaffold is in the workshop-tutorial-0 branch of this repo, so let's start by cloning it.

# Clone the repository
git clone https://github.com/LioQing/simple-rl-driver.git

# Checkout the workshop-tutorial-0 branch
git checkout workshop-tutorial-0

Next we want to install all the dependencies using Pip.

python -m pip install -r requirements.txt

The dependencies are as follows:

  • NumPy: For numerical operations and linear algebra.
  • Pygame: For game development.
  • Shapely: For geometric operations, specifically ray casting and collision detection.

Next, we are going to take a look at the implemented code in the branch.

Track Editor Mode

First of all, a complete track editor as already been implemented, which will let us draw any custom track with ease. It is implemented in the track module.

Try it with.

python main.py track -t my_track

Note

If my_track already exists, it will open the track and allow you to continue editing it.

All tracks are stored in data/tracks.

The controls are as follows:

Key Action
Press left mouse button Add a point
Hold and drag left mouse button Edit the curve
Press right mouse button Remove a point
Press Ctrl + S Save the track
Press Ctrl + Z, Del, Backspace, Esc Undo
Press Ctrl + H Flip the track horizontally
Press Ctrl + V Flip the track vertically
Press Ctrl + Q Quit the program

Utility Functions

First we will look at engine.utils which contains some useful functions.

  • rot_mat: Given the radian, returns a 2D rotation matrix, so multiplying a 2D vector with this will rotate the vector.
  • vec: Given the x and y coordinates, returns a 2D vector, it is returned in numpy.ndarray of shape (,2) representing a column vector so that it can be easily used with NumPy functions.
  • dir: Given the radian, returns a 2D unit vector representing the direction.
  • clamp: Given any value, the minimum and maximum allowed values, returns the clamped value.

Track Entity

Since the track editor is implemented, the track entity is also implemented in engine.entity.track.

Here are some of the most important things about it we should know, because we are going to use them in the coming tutorials:

  • Track.polyline: List of vectors forming the line segments of the track, which can be used to track car's progress.
  • Track.shapely_linear_ring: List of vectors forming the border of the track, which can be used to check if cars are within the track.
  • Track.get_start_dir: Returns unit vector representing which direction should cars be facing when the race starts.
  • Track.load: Loads the track given the name.
  • Track.draw: Draws the track given the screen and the camera, we will implement the camera in the next tutorial.

Activation Functions

In the engine.activations module are activation functions for the neural network. For now, not understanding them is fine, just know that they are used in neural network to keep the output values from each layer within a certain range while preserving the continuity of the function.

Gameplay and Training Mode

Part of game.main and train.main have been implemented.

The current implementation allows the program to consume command line arguments and pass them to the main_scene functions as args: argparse.Namespace, we will use it in future tutorials to allow users to pass in arguments.

Both modes also contains a load_nn function for loading saved neural network, and the training mode contains a save_nn function for saving neural networks.

Other Implemented Modules

Other implemented modules in the scaffold is not very important, here is a short description of what each of them do.

Clone this wiki locally