Reinforcement Learning: Twin Delayed Deep Deterministic Policy Gradient

[tareknaser]

Description

This pull request implements the TD3 (Twin Delayed Deep Deterministic Policy Gradient) algorithm, along with 2 test cases.

Implementation details

TD3 (Twin Delayed Deep Deterministic Policy Gradient) is a reinforcement learning algorithm designed for continuous action spaces. It builds upon DDPG and introduces twin critics and delayed updates to improve stability and performance.

Implemented 6 networks:

• policyNetwork (actor network)
• targetPNetwork (target actor network)
• learningQ1Network (first critic network)
• targetQ1Network (first target critic network)
• learningQ2Network (second critic network)
• targetQ2Network (second target critic network)

How Has This Been Tested?

• Included a Pendulum test that successfully passes with different configuration values.
  • With TargetNetworkSyncInterval = 1 ——> -1081.52
  • With TargetNetworkSyncInterval = 2 ——> -508.788
  • With TargetNetworkSyncInterval = 3 ——> -1209.31
• Additionally, added a test for continuous action spaces, which also passes.

The networks for the 2 tests are the same for DDPG and SAC for comparison.

[tareknaser]

I tested a python implementation of TD3 with 2 different networks (all other hyperparameters are the same) on gym Pendulum-v1 and these are the results I got

:

1- with 2 hidden layers
score: -723.92

2- with 3 hidden layers
score: -910.24

This is a similar situation to what I experienced in the unit tests.
But if I train them for longer I get very different results and the bigger network performs better. It’s a bit flaky. There are a lot of hyperparameters.
The unit tests we have use testAgent function which stops whenever the environment is solved. The unit tests are just meant to validate that the agent passes and the TD3 agent passes with different network architecture.
We can further train this in the examples repository.

[zoq]

All good from my side, tested it locally as well, without any test failures.

[mlpack-bot]

Second approval provided automatically after 24 hours. 👍