CppRl - PyTorch C++ Reinforcement Learning
CppRl is a reinforcement learning framework, written using the PyTorch C++ frontend.
It is very heavily based on Ikostrikov's wonderful pytorch-a2c-ppo-acktr-gail. You could even consider this a port. The API and underlying algorithms are almost identical (with the necessary changes involved in the move to C++).
It also contains an implementation of a simple OpenAI Gym server that communicates via ZeroMQ to test the framework on Gym environments.
CppRl aims to be an extensible, reasonably optimized, production-ready framework for using reinforcement learning in projects where Python isn't viable. It should be ready to use in desktop applications on user's computers with minimal setup required on the user's side.
At the time of writing, there are no general-use reinforcement learning frameworks for C++. I needed one for a personal project, and the PyTorch C++ frontend had recently been released, so I figured I should make one.
- Implemented algorithms:
- Recurrent policies (GRU based)
- Continuous control
- Discrete control
- Cross-platform compatibility (tested on Windows 10, Ubuntu 16.04, and Ubuntu 18.04)
- Solid test coverage
- Decently optimized (always open to pull requests improving optimization though)
An example that uses the included OpenAI Gym server is provided in
example. It can be run as follows:
It takes about 60 seconds to train an agent to 200 average reward on my laptop (i7-8550U processor).
The environment and hyperparameters can be set in
Note: The Gym server and client aren't very well optimized, especially when it comes to environments with image observations. There are a few extra copies necessitated by using an inter-process communication system, and then
gym_client.cpp has an extra copy or two to turn the observations into PyTorch tensors. This is why the performance isn't that good when compared with Python libraries running Gym environments.
CMake is used for the build system.
Most dependencies are included as submodules (run
git submodule update --init --recursive to get them).
Libtorch has to be installed seperately.
cd pytorch-cpp-rl mkdir build && cd build cmake .. make -j4
cd pytorch-cpp-rl mkdir build && cd build cmake -G "Visual Studio 15 2017 Win64" -DCMAKE_PREFIX_PATH=C:/path/to/libtorch .. cmake --build . --config Release
Before running, make sure to add
libtorch/lib to your
PATH environment variable.
Windows performance is about 75% that of Linux's at the moment. I'm looking into how to speed things up.
You can run the tests with
build/Release/cpprl_tests.exe on Windows).