grl
Generic Reinforcement Learning Library
Copyright 2015-2020 Wouter Caarls
This program is free software: you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation, either version 3 of the License, or (at your option) any later version.
This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with this program. If not, see http://www.gnu.org/licenses/.
Introduction
GRL is a C++ reinforcement learning library that aims to easily allow evaluating different algorithms through a declarative configuration interface.
Installation
Ubuntu 20.04
sudo apt install git cmake g++ libeigen3-dev libpython3-dev python3-distutils libz-dev
git clone https://github.com/wcaarls/grl.git
For the visualization, additionally install
sudo apt-get install libgl1-mesa-dev freeglut3-dev
For the configurator, additionally install
sudo apt-get install python3-yaml python3-tk
For the tensorflow addon, additionall install
Tensorflow 2
# Tensorflow C API 2.4.0
wget https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-2.4.0.tar.gz
sudo tar zxvf libtensorflow-gpu-linux-x86_64-2.4.0.tar.gz -C /usr/local
# CUDA Toolkit 11.0
sudo apt-key adv --fetch-keys http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64/7fa2af80.pub
sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/cuda/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda.list'
sudo bash -c 'echo "deb http://developer.download.nvidia.com/compute/machine-learning/repos/ubuntu1804/x86_64 /" > /etc/apt/sources.list.d/cuda_learn.list'
sudo apt update
sudo apt install cuda-11-0
sudo apt install libcudnn8
Then edit ~/.bashrc
to include the following, and open a new terminal
export PATH=$PATH:/usr/local/cuda-11.0/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-11.0/lib64
Install Tensorflow 2 for Python
sudo -H python3 -m pip install tensorflow
Tensorflow 1
Tensorflow 1 is deprecated. Consider swithing to Tensorflow 2 (see above).
# Tensorflow C API 1.14.0
wget https://storage.googleapis.com/tensorflow/libtensorflow/libtensorflow-gpu-linux-x86_64-1.14.0.tar.gz
sudo tar zxvf libtensorflow-gpu-linux-x86_64-1.14.0.tar.gz -C /usr/local
# CUDA Toolkit 10.0
wget https://developer.nvidia.com/compute/cuda/10.0/Prod/local_installers/cuda_10.0.130_410.48_linux
wget http://developer.download.nvidia.com/compute/cuda/10.0/Prod/patches/1/cuda_10.0.130.1_linux.run
chmod a+x *.run
sudo ./cuda_10.0.130_410.48_linux --override # When asked, skip installing graphics driver
sudo ./cuda_10.0.130.1_linux.run
# cuDNN 7.6.5 for CUDA 10.0
# First log in with NVIDIA developer account
# Download https://developer.nvidia.com/compute/machine-learning/cudnn/secure/7.6.5.32/Production/10.0_20191031/cudnn-10.0-linux-x64-v7.6.5.32.tgz
sudo tar zxvf cudnn-10.0-linux-x64-v7.6.5.32.tgz -C /usr/local
Then edit ~/.bashrc
to include the following, and open a new terminal
export PATH=$PATH:/usr/local/cuda-10.0/bin
export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-10.0/lib64
Install Python 3.7 from ppa or from source, e.g.
sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt-get update
sudo apt-get install python3.7 python3.7-dev
Install Tensorflow for Python 1.14.0
sudo -H python3.7 -m pip install tensorflow==1.14.0 keras==2.2.4
Finally, edit the network files (cfg/4layer_*.py
) to use the correct Python version if
necessary.
If there are errors relating to Python loading
libtensorflow_framework.so.1
, it is because it comes with its own version,
and it finds the version from the C API. Consider adding the path of the
Python version to LD_LIBRARY_PATH
, before the C version.
Ubuntu 16.04
sudo apt install git cmake g++ libeigen3-dev libpython3-dev python3-distutils libz-dev
git clone https://github.com/wcaarls/grl.git
For the visualization, additionally install
sudo apt-get install libgl1-mesa-dev freeglut3-dev
For the odesim environment, additionally install
sudo apt-get install libqt4-opengl-dev libtinyxml-dev libmuparser-dev libode-dev
For the RBDL environment and MUSCOD integration, additionally install
sudo apt-get install liblua5.1-dev
For the configurator, additionally install
sudo apt-get install python-yaml python-tk
For the zeromq addon, additionally install
sudo apt-get install libprotoc-dev protobuf-compiler libsodium-dev
Then download a recent (> 4.0.0) version of zeromq and run './configure, make, make install'
For the tensorflow addon, additionally install
sudo apt-get install libprotoc-dev protobuf-compiler
Then download TensorFlow for C 1.14.0 from https://www.tensorflow.org/install/install_c in a place that can be found by the compiler and runtime environment. To use Python to build the networks, the same Tensorflow version (and Keras 2.2.4) should be available under Python as well.
Setup
mkdir build
cd build
cmake ..
make
Running
To directly perform an experiment, run
./grld ../cfg/pendulum/sarsa_tc.yaml
To start the configurator instead, run
cd ../bin
./grlc ../cfg/pendulum/sarsa_tc.yaml
Visualizations
GRL comes with standard visualizations for value functions, policies, and the integrated environments (e.g. pendulum swing-up, cart-pole swing-up, compass walker)
Further reading
See grl.pdf