Reinforcement Learning Project

Repository for the project of the course "Reinforcement Learning" of the master degree "Data Science and Scientific Computing" @Units

About the project

The aim of the project is to teach a computer agent to play the Atari 2600 game Breakout via the Reinforcement Learning technique of Double (Deep) Q-Learning.

We will use images of the screen game to make our agent learn a policy that can allow it to score a sufficient number of points in the game (how many depends on the computing power and the time at your disposal).

You can find the slides for the presentation here.

The code

The notebooks files all have a corresponding .py file for the versioning (otherwise in the .ipynb files it's impossible to see the changes through git). They are made or by downloading the notebook as a .py file or through the Jupyter plugin jupytext, that reads and writes notebooks as plain text files: either Python, Julia, R scripts, Markdown, or R Markdown documents.

• The notebook Breakout_intro.ipynb is used to understand the environment of the Breakout game.

• The notebook Breakout.ipynb is used to train the agent, and saves at specific checkpoints the weights of the neural networks and the statistics of the game (episode duration, rewards, loss).

• The notebook Breakout_resume.ipynb is used to resume the training of the agent in case it stops (I noticed that it usually uses up all the available memory and it crashes). It uses automatically the last saved checkpoint and saves new ones.

• The notebook Breakout_play.ipynb is used to play a game with the agent. Depending on how much you want your agent to be trained, you can use different checkpoint weights.

• The file Breakout_script.py is a cleaned version of the notebook Breakout.ipynb to be used in a cluster. It can be submitted as a job in the cluster using the script script_cluster.sh.

• The file Breakout_resume_script.py is a cleaned version of the notebook Breakout_resume.ipynb to be used in a cluster. It can be submitted as a job in the cluster using the script script_cluster_resume.sh.

Info

To kill the annoying windows of the environment rendering that don't close, open a terminal and digit:

xkill

then click on the windows you want to kill.

Linux Setup

Download, clone or fork (your choice) this repository in a directory PATH_TO_DIR/, then enter in the folder using:

cd PATH_TO_DIR/ReinforcementLProject/code

Create a virtual environment using python3 (commands are provided for Debian-like GNU/Linux distributions --- see https://docs.python.org/3/library/venv.html for further explanation). If you don't have installed both pip and virtualenv, run

# Install pip for Python 3:
sudo apt-get install python3-pip
# Install virtualenv
python3 -m pip install virtualenv

If you have, skip the previous commands.

Then run

python3 -m virtualenv -p "$(which python3)" venv

If the following error

$ python3 -m virtualenv -p "$(which python3)" venv
ImportError: cannot import name 'ensure_file_on_disk'

appears, it is because of multiple versions of virtualenv are installed. Remove all the versions using pip3 uninstall virtualenv multiple times, until the message Cannot uninstall requirement virtualenv, not installed appears. Then re-run

python3 -m pip install --user virtualenv
python3 -m virtualenv -p "$(which python3)" venv

Now you should see PATH_TO_DIR/venv/ folder. Activate the environment and install the requirements (don't specify the versions of the packages, so you'll get the latest versions):

source venv/bin/activate
python3 -m pip install -r ./requirements_INSTALLER.txt

Register the just-installed virtual environment for use with Jupyter:

python3 -m ipykernel install --user --name ReinforcementL --display-name "Python3 (RL virtualenv)"

Then type:

pip freeze

and save the output in a file named requirements.txt.

Too see the dependencies of a package, run

pip3 show <name_package>

Open your notebooks using jupyter-notebook (or jupyter-lab):

python3 -m jupyter notebook

To deactivate the environment use deactivate command.

To convert the .ipynb file in HTML use the following command

jupyter nbconvert --to html notebook.ipynb

Breakout

If you want to use Gym Atari environments, among which there is Breakout,

source venv/bin/activate
pip install gym[atari]

ROMs

In order to import ROMS, you need to download Roms.rar from the Atari 2600 VCS ROM Collection and extract the .rar file. Save this file in a folder called ROMs.

To do so, open a terminal and install unrar:

sudo apt-get install unrar

Then to extract the file run:

unrar e -r Roms.rar

You'll get two zip files: ROMS.zip and HC ROMS.zip. Using the archive manager, unzip both files in the current directory.

Once you've done all that, run:

python -m atari_py.import_roms <path to folder>

In my case, this is:

python -m atari_py.import_roms ~/Documenti/Reinforcement\ Learning/Project/code/ROMs/ROMS/

This should print out the names of ROMs as it imports them. The ROMs will be copied to your atari_py installation directory (in my case: /home/angela/Documenti/Reinforcement Learning/Project/code/venv/lib/python3.6/site-packages/atari_py/atari_roms/).

Breakout's ROM

If you only want to use the game Breakout, you can only keep one ROM in the folder ROMs:

Breakout - Breakaway IV (Paddle) (1978) (Atari, Brad Stewart - Sears) (CX2622 - 6-99813, 49-75107) ~.bin

and delete the other files. Then opening a terminal in that folder run

python -m atari_py.import_roms ./ROMs