This is my implementation of the discrete Soft Actor Critic algorithm for Reinforcement Learning. More details about the algorithm can be found here:
- Christodoulou, Petros. "Soft actor-critic for discrete action settings." arXiv preprint arXiv:1910.07207 (2019).
This implementation works with all the Open AI Gym Environments having a discrete action set, but right now it plays Space Invaders.
The requirements for the repo to work are in env.yml (if you use conda: conda env create -f env.yml).
First, edit the config01.json file (or create a new one), with the metaparameters and variables used for the training:
-
configId: string identifier of the traning,
-
env_parameters:
- screen_size: size of the video game screen,
- frame_skip: frame skip value,
- seed_value: random number generators' seed
-
training_parameters:
- n_episodes: number of episodes to play during training
- t_tot_cut: maximum number of moves to play in each episode
- batch_size: number of previous moves to use during a training step
-
agent_parameters:
- gamma: discount rate
- lr_Q: learning rate of the Q-function
- lr_pi: learning rate of the policy
- lr_alpha: learning rate of the temperature
- alpha: value of the temperature. Set it to "auto" if you want it to be learnt during training
- tau: update rate of the Q target functions
- entropy_rate: constant to be put in front of the target entropy if alpha="auto". The lower, the less random the moves will be at the end.
- h_dim: hidden units of the Q-functions flat layers
- h_mu_dim: hidden units of the policy flat layers
The run "python train.py" to start training. The training curve will be saved into the "train_figs" folder every 100 episodes, while the model parameters will be stored in "saved_models".
When the training is over, launch "python best_of_100_episodes.py" to have the best episode reward out of a sample of 100. Best score on the Open AI Leader Board (https://github.com/openai/gym/wiki/Leaderboard) is 3454. Here, we should get around 1800.
To generate a small movie of a match use instead "python generate_match_video.py". There is an example in the /video directory.