Integration of the skrl RL library

[Toni-SM]

Description

Integration of the skrl reinforcement learning library

• Adds a wrapper function to wrap an IsaacEnv for skrl compatibility
• Adds a trainer class that also logs episode information
• Adds train.py and play.py in workflows to run skrl
• Adds configuration files for currently included environments with parameters matching that of rl_games as close as possible

Dependencies:

• skrl >= 0.10.0: https://github.com/Toni-SM/skrl

Type of change

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Checklist

• I have run the pre-commit checks with pre-commit run --all-files (see here instructions to set it up)
• I have made corresponding changes to the documentation
• My changes generate no new warnings
• I have added tests that prove my fix is effective or that my feature works
• I have updated the changelog and the corresponding version in the extension's config/extension.toml file

[Mayankm96]

Thanks a lot for the pull request! I'll take a look at it tomorrow.

Can you please also update the changelog and the version in the config/extension.toml file of omni.isaac.orbit_envs?

https://isaac-orbit.github.io/orbit/source/refs/contributing.html#maintaining-a-changelog

[Toni-SM]

Done, CHANGELOG and extension.toml files updated!!!

[Mayankm96]

Hi!

I finally got time today to test this out. I am yet to try training on the classic environments but would be nice to do some evaluations while having the same configurations for PPO across the frameworks.

Right now, mostly tried training for the reach environment. I have the following comments.

Issues

1. Currently, while training the reach environment, I am seeing a divergence in the loss curve of the value function. Is it possible the parameters are not tuned? I suppose there might be tiny differences in algorithmic implementations but don't think the value function should be diverging. One possible explanation could be that the learning iterations are too large and that's causing some kind of collapse.

   

2. The learned policy for reach seems to be not that stable (I tried the best_agent.pt) . It is quite shaky which I don't see when I train with rsl_rl. Maybe again the parameters are not similar or the same and that is the potential problem.

   skrl.mp4

3. Currently, the play.py is using the runner class to run the agent-env interaction.

   • When I hit the STOP icon on the GUI, it just freezes and doesn't exit the script quickly. Is there a possible issue in the trainer that doesn't let it exit?
   • When I hit the X icon on the GUI, I observe a similar behavior.

Nice to haves

• Is it possible to also log the episode information in the runner? Many times in robotics, we care about how each reward term is evolving (for weight tuning). The cumulative reward is often hard to decipher when having many reward terms
• Can the play.py just load the last or best checkpoint by default? We do something similar for rsl-rl (code snippet).
• An updated example on how to use a learned policy outside the runner loop. It should usually be simple since that is a torch model that needs to be loaded. But it would be great if we can have an example to show this. It comes quite handy when deploying a policy in a complete robot system (in this case, an extension).

Additional comments

• We will also need to add the license of skrl inside the docs/licenses/dependencies directory for book-keeping purposes.

[Toni-SM]

Hi @Mayankm96

---

Comments:

1. Loss curve divergence of the value function should not be a problem. rl_games also diverge... I checked the rsl_rl PPO implementation and there is a small difference in the way KL and learning rate scheduler are computed, also some training parameters are different

   

2. I mapped (as far as possible) the same parameters as .yaml files for rl_games. The stability of the policy comes mostly from the number of time steps used compared to rsl_rl.

   • skrl: 8000,
   • rl_games: 16 * 500 = 8000
   • rsl_rl: 64 * 250 = 16000

   rls_rl works just as "badly" as the others at 8000 timesteps.
   Training for 16000 timesteps generates a better curves

   

3. Solved

---

Nice to haves

1. Log the episode information:
   Done

2. Load the last or best checkpoint by default:
   Done

3. An updated example of how to use a learned policy outside the runner loop:
   It sounds great. I am going to create an example for futures pull requests

---

Additional comments

1. skrl license in docs:
   Done

[Mayankm96]

Thanks a lot for providing more information. I just added some minor feedback on the code to understand things a bit better. Once that is cleared up, we can merge this awesome PR :)

[Toni-SM]

Done...

Summary:

• Create a customized trainer in the file where the skrl wrapper is defined for:
  • log episode information during training
  • simplify train.py and play.py
• Remove logging in play.py
• Move the import statements to the beginning of the config.py file

[Mayankm96]

The PR looks good to me now. Thanks a lot for working on skrl integration. It is great to have more libraries supported in the workflows.

Would love to try out other agents as well in skrl at some point and provide examples :)

[Toni-SM]

Great!
Let's work on integrating new agents in future pull requests!

By the way... The online documentation does not reflect any changes 🤔

[Mayankm96]

It should be up now! :)

By the way, I made some minor refactoring in this commit f7e4183 :

• renaming of SkrlLogTrainer to SkrlSequentialLogTrainer
• readding of the eval mode to the class for completeness
• some docstring fixes
• setting the agent to "eval" mode in the play.py script

[Toni-SM]

Nice... It looks much better now