torch-twrl: Reinforcement Learning in Torch
torch-twrl is an RL framework built in Lua/Torch by Twitter.
git clone https://github.com/torch/distro.git ~/torch --recursive cd ~/torch; bash install-deps; ./install.sh
git clone --recursive https://github.com/twitter/torch-twrl.git cd torch-twrl luarocks make
Want to play in the gym?
Start a virtual environment, not necessary but it helps keep your installation clean
pip install virtualenv virtualenv venv source venv/bin/activate pip install gym pip install -r src/gym-http-api/requirements.txt brew install ffmpeg
Works so far?
You should have everything you need:
- Start your gym_http_server with
- In a new console window (or tab), run the example script (policy gradient agent in environment CartPole-v0)
cd examples chmod u+x cartpole-pg.sh ./cartpole-pg.sh
This script sets parameters for the experiment, in detail here is what it is calling:
th run.lua \ -env 'CartPole-v0' \ -policy categorical \ -learningUpdate reinforce \ -model mlp \ -optimAlpha 0.9 \ -timestepsPerBatch 1000 \ -stepsizeStart 0.3 \ -gamma 1 \ -nHiddenLayerSize 10 \ -gradClip 5 \ -baselineType padTimeDepAvReturn \ -beta 0.01 \ -weightDecay 0 \ -windowSize 10 \ -nSteps 1000 \ -nIterations 1000 \ -video 100 \ -optimType rmsprop \ -verboseUpdate true \ -uploadResults false \ -renderAllSteps false
Your results should look something our results from the OpenAI Gym leaderboard
- Test the gym-http-api
cd /src/gym-http-api/ nose2
- Start a Gym HTTP server in your virtual environment
- In a new console window (or tab), run torch-twrl tests
luarocks make; th test/test.lua
Testing of RL development is a tricky endeavor, it requires well established, unified, baselines and a large community of active developers. The OpenAI Gym provides a great set of example environments for this purpose. Link: https://github.com/openai/gym
The OpenAI Gym is written in python and it expects algorithms which interact with its various environments to be as well. torch-twrl is compatible with the OpenAI Gym with the use of a Gym HTTP API from OpenAI; gym-http-api is a submodule of torch-twrl.
All Lua dependencies should be installed on your first build.
Note: if you make changes, you will need to recompile with
torch-twrl implements several agents, they are located in src/agents. Agents are defined by a model, policy, and learning update.
- model: noModel
- policy: random
- learningUpdate: noLearning
- model: qFunction
- policy: egreedy
- learningUpdate: tdLambda - implements temporal difference (Q-learning or SARSA) learning with eligibility traces (replacing or accumulating)
- Policy Gradient Williams, 1992:
- model: mlp - multilayer perceptron, final layeer: tanh for continuous, softmax for discrete
- policy: stochasticModelPolicy, normal for continuous actions, categorical for discrete
- learningUpdate: reinforce
Important note about agent/environment compatibility:
The OpenAI Gym has many environments and is continuously growing. Some agents may be compatible with only a subset of environments. That is, an agent built for continuous action space environments may not work if the environment expects discrete action spaces.
Here is a useful table of the environments, with details on the different variables that may help to configure agents appropriately.
Continuous integration is accomplished by building with Travis. Testing is done with LUAJIT21, LUA51 and LUA52 with compilers gcc and clang.
Tests are defined in the /tests directory with separate basic unit tests set and a Gym integration test set.
- LUA52 and libhash not working, so tilecoding examples fail in LUA52.
- Automatic policy differentiation with Autograd
- Parallel batch sampling
- Additional baselines for advantage function computation
- Cross Entropy Method (CEM)
- Deep Q Learning (DQN)
- Double DQN
- Asynchronous Advantage Actor-Critic (A3C)
- Deep Deterministic Policy Gradient (DDPG)
- Trust Region Policy Optimization (TRPO)
- True Online-TD
- Boyan, J., & Moore, A. W. (1995). Generalization in reinforcement learning: Safely approximating the value function. Advances in neural information processing systems, 369-376.
- Sutton, R. S. (1988). Learning to predict by the methods of temporal differences. Machine learning, 3(1), 9-44.
- Singh, S. P., & Sutton, R. S. (1996). Reinforcement learning with replacing eligibility traces. Machine learning, 22(1-3), 123-158.
- Barto, A. G., Sutton, R. S., & Anderson, C. W. (1983). Neuronlike adaptive elements that can solve difficult learning control problems. Systems, Man and Cybernetics, IEEE Transactions on, (5), 834-846.
- Sutton, Richard S., and Andrew G. Barto. Reinforcement learning: An introduction. Vol. 1. No. 1. Cambridge: MIT press, 1998.
- Williams, Ronald J. "Simple statistical gradient-following algorithms for connectionist reinforcement learning." Machine learning 8.3-4 (1992): 229-256.