PongDQN

Introduction

Experimental implementaion of the Double DQN algorithm in C++ from scratch to train it on custom Pong environment. DQN agent learns to play Pong by playing against computer opponents using inputs from RAM. Pong environment is rendered using OpenGL with Win32 API for window and context creation. DQN library uses a simple neural network library implimented from scratch, a custom math library supporting vectors and matrices.

DQN agent learns to hold its own against chaser AI even if max vertical speed of paddle is less than max vertical speed of ball in about 400 episodes (approx 1 min on i5 11th gen at 3 GHz).

Results

Training Log	Testing Log

Running demonstration

A x86-64 binary is provided in bin folder for demonstration purpose. After running this library a pre-trained DQN agent (left) will play with a simple chaser algorithm (right). User can use controls specified in this document to control demo.

Controls

1. M
To speed up the environment.
2. L
To log the model as cache.
3. P
Pause the application. To continue press enter on terminal. (Application uses blocking I/O call for pausing instead of busy waiting)

Config

Before compiling application their are some configurations that should be made in main.cpp according to the needs.

1. OPPONENT AI
1. BOT_CHASER
A simple chaser AI as opponent on DQN agent. It alway tries to follow ball to hit at center of paddle.
2. BOT_EDGE
Slightly chalanging AI which has 50-50% probability to hit ball at center or hitting ball at edge.
3. PLAYER
Player can play as opponent using mouse.
4. BOT_AGENT
A second pre-trained DQN agent which can be used to play against agent on left.
2. TRAIN
If you want to train the model uncomment the define statement.
3. MODEL_ID
ID of model, which will be used in filename of model saved during traing,training log and testing log.

You can change model to be loaded for each agent in main.cpp. (Search for comment "Load model")

Building code

(This project only supports windows as UI and OpenGL initialization uses Windows APIs)

• using vscode: After downloading repo simply open PongDQN folder in vscode. After adjusting configurations according to need simply run the task provided in .vscode/task.json. Binary will be created in bin folder.

Modules

This repository contains 6 libraries that are implemented from scratch in C++:

1. Mathlib:
A simple math library containing implementation of vectors and matrix which is used to implement neural networks.
2. Neurallib:
A simple neural network library containing implementation of deep neural networks, activation functions and loss functions which is used to implement DQN agent.
3. Gymlab:
A simple gym environment library for RL agents containing implementation of PONG environment which is used alongside DQN agent to train it.
4. Uilib:
A simple UI library(windows) for creating windows which can be used for rendering.
5. Renderlib:
A simple rendering library for rendering gym environment for human understanding. It provides way to draw simple 2d shapes using OpenGL API.
6. Rllib:
A simple reinforcement learning library containing implementation of DQN algorithm.

Screenshots

Pong environment
Training log
Test log