Julia code for the book Reinforcement Learning An Introduction
Clone or download
Fetching latest commit…
Cannot retrieve the latest commit at this time.
Type Name Latest commit message Commit time
Failed to load latest commit information.
docs modify params Jan 17, 2019
src modify params Jan 17, 2019
.gitignore ignore Manifest.toml Jan 2, 2019
LICENSE Happy New Year Dec 31, 2018
Project.toml Happy New Year Dec 31, 2018
README.md update README Jan 17, 2019



"To think is to forget a difference, to generalize, to abstract."

Jorge Luis Borges, Funes the Memorious

This repo reproduces almost all the figures on the book Reinforcement Learning: An Introduction(2nd).



Just run the following command to install this package:

$ julia -e "using Pkg; Pkg.add(\"Plots\"); Pkg.add(PackageSpec(url=\"https://github.com/Ju-jl/Ju.jl.git\")); Pkg.add(PackageSpec(url=\"https://github.com/Ju-jl/ReinforcementLearningAnIntroduction.jl\"));"

Then enter the REPL:

julia> using RLIntro  # Hold on! It might take several minutes to pre-compile

julia> @show [f for f in names(RLIntro) if startswith(string(f), "fig")];  # list all the functions to reproduce corresponding figures
[f for f = names(RLIntro) if startswith(string(f), "fig")] = Symbol[:fig_10_1, :fig_10_2, :fig_10_3, :fig_10_4, :fig_10_5, :fig_11_2, :fig_12_3, :fig_13_1, :fig_13_2, :fig_2_1, :fig_2_2, :fig_2_3, :fig_2_4, :fig_2_5, :fig_2_6, :fig_3_2, :fig_3_5, :fig_4_1, :fig_4_2, :fig_4_3, :fig_5_1, :fig_5_2, :fig_5_3, :fig_5_4, :fig_6_2_a, :fig_6_2_b, :fig_6_2_c, :fig_6_2_d, :fig_6_3_a, :fig_6_3_b, :fig_6_5, :fig_7_2, :fig_8_2, :fig_8_4, :fig_8_4_example, :fig_8_5, :fig_8_7, :fig_8_8, :fig_9_1, :fig_9_10, :fig_9_2_a, :fig_9_2_b, :fig_9_5]

julia> fig_2_2()  # reproduce figure_2_2

Notice that for some figures you may need to install pdflatex.


If you would like to make some improvements, I'd suggest the following workflow:

  1. Clone this repo and enter the project folder.
  2. Enter the pkg mode and (RLIntro) pkg> add https://github.com/Ju-jl/Ju.jl.git (Because the Ju.j is not registered yet. It will not be a big problem after Julia 1.1 get released)
  3. Make changes to some existing Environment or create a new Environment and include it in the REPL (like include("src/environments/MultiArmBandits.jl"))
  4. Make changes to the related source codes and include it in the REPL (like include("src/chapter02/ten_armed_testbed.jl"))
  5. Run the functions to draw figures (fig_2_2()).
  6. Repeat the above three steps.


Chapters Figures Description
Chapter01 Run play() will prompt an interactive interface to play the TicTacToe game.
Chapter02 fig_2_2, fig_2_3, fig_2_4, fig_2_5, fig_2_6
Chapter03 fig_3_2, fig_3_5 Here the heatmap is used to represent the value
Chapter04 fig_4_1, fig_4_2_policy, fig_4_2_value, fig_4_3
Chapter05 fig_5_1_no_usable_ace_n_10000, fig_5_1_usable_ace_n_10000 Warning!!! The result is different to the figures on the book. Please help to correct it.
fig_5_2_no_usable_ace_n_500000, fig_5_2_usable_ace_n_500000, fig_5_2_no_usable_ace_policy_n_500000, fig_5_2_usable_ace_policy_n_500000
fig_5_3, fig_5_4
Chapter06 fig_6_2_a, fig_6_2_b, fig_6_2_c, fig_6_2_d
fig_6_3_a, fig_6_3_b
Chapter07 fig_7_2
Chapter08 fig_8_2, fig_8_4_example, fig_8_4
fig_8_5, fig_8_7, fig_8_8_a, fig_8_8_b
Chapter09 fig_9_1_a, fig_9_1_b, fig_9_2_a, fig_9_2_b, fig_9_5
Chapter10 fig_10_1_1, fig_10_1_2, fig_10_1_3, fig_10_1_4, fig_10_1_5
fig_10_3, fig_10_4, fig_10_5
Chapter11 fig_11_2
Chapter12 fig_12_3 Other figures in Chapter12 are not that easy to reproduce by using the Ju.jl package. You may take a try and correct me with a PR.
Chapter13 fig_13_1, fig_13_2 Figure_13_2 is a slightly different to the original figure on the book. Thanks to Eric Graves's clarification, fixed in #3

Related Packages