Add more reinforcement learning tutorials

[wittyicon29]

Fixes #1280

Description

RL_tutorial_PyTorch.zip

Trust Region Policy Optimization (TRPO) is a policy optimization algorithm for reinforcement learning. It aims to find an optimal policy by iteratively updating the policy parameters to maximize the expected cumulative reward. TRPO addresses the issue of unstable policy updates by imposing a constraint on the policy update step size, ensuring that the updated policy stays close to the previous policy.

The code begins by importing the necessary libraries, including PyTorch, Gym (for the environment), and the Categorical distribution from the PyTorch distributions module.

Next, the policy network is defined using a simple feed-forward neural network architecture. The network takes the state as input and outputs a probability distribution over the available actions. The network is implemented as a subclass of the nn.Module class in PyTorch.

The trpo function is the main implementation of the TRPO algorithm. It takes the environment and policy network as inputs. Inside the function, the state and action dimensions are extracted from the environment. The optimizer is initialized with the policy network parameters and a learning rate of 0.01. The max_kl variable represents the maximum allowed Kullback-Leibler (KL) divergence between the old and updated policies.

The surrogate_loss function calculates the surrogate loss, which is used to update the policy. It takes the states, actions, and advantages as inputs. The function computes the log probabilities of the selected actions using the current policy. It then calculates the surrogate loss as the negative mean of the log probabilities multiplied by the advantages. This loss represents the objective to be maximized during policy updates.

The update_policy function performs the policy update step using the TRPO algorithm. It takes a trajectory, which consists of states, actions, and advantages, as input. The function performs multiple optimization steps to find the policy update that satisfies the KL divergence constraint. It computes the surrogate loss and the KL divergence between the old and updated policies. It then performs a backtracking line search to find the maximum step size that satisfies the KL constraint. Finally, it updates the policy parameters using the obtained step size.

The main training loop in the trpo function runs for a specified number of epochs. In each epoch, a trajectory is collected by interacting with the environment using the current policy. The trajectory consists of states, actions, and rewards. The advantages are then calculated using the Generalized Advantage Estimation (GAE) method, which estimates the advantages based on the observed rewards and values. The update_policy function is called to perform the policy update using the collected trajectory and computed advantages.

After each epoch, the updated policy is evaluated by running the policy in the environment for a fixed number of steps. The total reward obtained during the evaluation is printed to track the policy's performance.

To use the code, an environment from the Gym library is created (in this case, the CartPole-v1 environment). The state and action dimensions are extracted from the environment, and a policy network is created with the corresponding dimensions. The trpo function is then called to train the policy using the TRPO algorithm.

Make sure to provide additional explanations, such as the concepts of policy optimization, the KL divergence constraint, the GAE method, and any other relevant details specific to your tutorial's scope and target audience.

Checklist

• The issue that is being fixed is referred in the description (see above "Fixes New Tutorial : Add more Reinforcement Learning Tutorials  #1280 ")
• Only one issue is addressed in this pull request
• Labels from the issue that this PR is fixing are added to this pull request
• No unnecessary issues are included into this pull request.
• cc @vmoens @nairbv
• @docathon-h1-2023 #docathon-h1-2023

[netlify]

✅ Deploy Preview for pytorch-tutorials-preview ready!

Name	Link
🔨 Latest commit	527daed
🔍 Latest deploy log	https://app.netlify.com/sites/pytorch-tutorials-preview/deploys/647caa173d64720008d26cd4
😎 Deploy Preview	https://deploy-preview-2422--pytorch-tutorials-preview.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.

[vmoens]

Hello! Thanks a lot for contributing.
A few high-level comments before I dig into the code:

• Can you add your tutorial to https://github.com/pytorch/tutorials/blob/main/index.rst?
• Can you follow the template here
• What is the cuda extension dispatcher you provide?
• Your files do not have an extension (eg .py) and are not executable (comments should be commented out etc).

Feel free to look at other tutorials first to get a sense of what the script should look like, eg here.

[wittyicon29]

Sure, I'll add my tutorial to https://github.com/pytorch/tutorials/blob/main/index.rst along with executable extension according to the template.
The cuda extension dispatcher wasn't supposed to be included , it's a mistake from my side.
Extremely sorry for the inconvenience.