A2C cartpole benchmark

[lgraesser]

Experiment Result

See PR #180 for an example

The data files to upload below are all created automatically in the data/ folder.

Github does not support .json and .csv upload, but .txt. Please rename those files .txt before uploading.

Abstract

Benchmark experiment for A2C on the Cartpole environment

Methods

• generalized advantage estimation for the advantage
• random search to sweep hyper-parameters

To Reproduce

1. JSON spec: a2c_gae_mlp_separate_cartpole_spec.txt
2. git SHA (contained in the file above): 7254254

Results

All the results contributed will be added to the benchmark, and made publicly available on Dropbox.

• 1. full experiment data zip: (please find our contact in README and request a "Dropbox file request" to upload it to the public benchmark folder.)
• 2. experiment graph:
  
• 3. max fitness score and experiment_df: 1.20, a2c_gae_mlp_separate_cartpole_experiment_df.txt
• 4. best trial JSON spec: a2c_gae_mlp_separate_cartpole_t33_spec.txt
• 5. best trial graph:
  
• 6. best session graph (optional):
  

Discussion (optional)

Looking at the experiment graph, we can tell some effects from the hyperparameters. Going from left to right columns:

1. entropy_coef (this encourages exploration with larger value): the larger the entropy coef, the slower the convergence. For such a simple environment as cartpole, agent does not need much exploration, so low coef leads to faster learning. Some entropy is still beneficial. Convergence speed peaks at entropy_coef=0.02
2. lambda (of GAE): performance is not sensitive to it, but high value can cause some drop in strength.
3. training_frequency (per episode for OnPolicyReplay): as seen, training cannot happen too frequently (high variance) or infrequently (slow learning). There is an optimal frequency for maximum convergence speed, which is 2 for this experiment.
4. hid_layers_activation: tanh works the best overall.
5. lr_decay_frequency (lr = learning rate): the strength is concentrated at high value for frequency at 2k; the speed graph has a peak at 2k (one outlier point); the stability graph has a noticeable bump peaking at 4k to 6k. Overall it seems that if decay frequency is too high, learning rate drops very low very quickly, and learning slows down. If lr drops too slow, lr stays high, so learning is unstable. There is an optimal point in between. The effect can be quite noisy.