Add mountain car reward heatmap and reward-vs-time plotters

[shwang]

Putting the Heatmap / reward vs time plotting code from my ipynb into a python file.

Not sure if experiments/ is the best place to put this. Maybe imitation.scripts is better since the user might want to import these plotting functions into another file.

While translating the code from one format to another, I found some bugs, related yet again to VecNormalize. In this case it seems like my reward functions might not have actually received normalized inputs.

So I'd expect heatmaps to look different upon a new run.

[codecov]

Codecov Report

Merging #177 into master will increase coverage by 2.25%.
The diff coverage is 98.87%.

@@            Coverage Diff             @@
##           master     #177      +/-   ##
==========================================
+ Coverage   85.63%   87.89%   +2.25%     
==========================================
  Files          64       67       +3     
  Lines        4519     4683     +164     
==========================================
+ Hits         3870     4116     +246     
+ Misses        649      567      -82     

Impacted Files	Coverage Δ
src/imitation/analysis/mountain_car_plots.py	98.07% <98.07%> (ø)
src/imitation/rewards/common.py	100.00% <100.00%> (ø)
src/imitation/rewards/serialize.py	100.00% <100.00%> (ø)
src/imitation/util/reward_wrapper.py	89.58% <100.00%> (-0.22%)	⬇️
src/imitation/util/rollout.py	95.70% <100.00%> (ø)
tests/test_density_baselines.py	100.00% <100.00%> (ø)
tests/test_mountain_car_plots.py	100.00% <100.00%> (ø)
... and 11 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 96d9f8d...19c798e. Read the comment docs.

[AdamGleave]

Overall good. One change I'd suggest in your style is using explicit matplotlib Figure and Axes objects, rather than depending on the global state in matplotlib. This is more flexible and less error prone. Otherwise fairly minor comments.

[shwang]

Thanks for the review. I've unsuccessfully attempted before to figure out how to gracefully use the OOP interface to matplotlib, so it was good to learn a cleaner way to manage these plots.

It's a pain for me to manually check whether plots are working so I'm going to write a smoke test of some sort. Ask for another round of review after then.

[shwang]

I moved some reward-related helper functions and util.reward_wrapper.RewardFn into a new submodule imitation.rewards.common.

To make the functions importable and testable, I moved mountain_car_plots.py from experiments/ to a new submodule imitation.analysis.mountain_car_plots. (Also considered imitation.scripts, but this isn't really a script right now, more like a collection of plotting tools.)

This should be ready for another round of review now.

[shwang]

Somehow, this branch is set off the Sacred race condition that is fixed by IDSIA/sacred#473 4 out of 5 times I reran tests.

Maybe this is bad luck on my part (in one of the reruns, train_experts.sh was the breaking script, in another rerun train_experts.sh was fine but transfer_learn...sh was broken.); maybe this is because CircleCI somehow changed.

[shwang]

I think it would be really helpful for me if Sacred configs were picklable. This would allow us to upgrade to a version of Sacred that doesn't run into the FileStorageObserver race condition when I'm running local experiments, and let me use the same version of Sacred that is used by the Minecraft project.

Asked on IDSIA/sacred#508 if all they need is just dropping in pyrsistent.PMap.

[AdamGleave]

LGTM, new OO-style for matplotlib much easier to understand, and +1 for tests.

There's a few minor changes I've suggested but coding as approve since no need for me to review again.