[S-02-1] Understanding HAC & UOF theory and sources

[CUN-bjy]

Ref)

• HAC
  • Levy, Andrew, et al. "Learning multi-level hierarchies with hindsight." arXiv preprint arXiv:1712.00948 (2017).
  • code : Hierarchical-Actor-Critic-HAC-PyTorch
• UOF,
  • Yang, Xintong, et al. "Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation." IEEE Transactions on Neural Networks and Learning Systems (2021).
  • code : UOF-paper-code

[CUN-bjy]

1. HAC(Hierarchical Actor-Critic)

: Learning multi-level hierarchies with hindsight

Features

• DDPG + HER for each-level policy
• Nested Policy Structure
• parallel training for multi-level policy
  • hindsight action transition
  • hindsight goal transition
  • subgoal testing transition

2. UOF(Universal Option Framework)

: Hierarchical Reinforcement Learning With Universal Policies for Multistep Robotic Manipulation

Features

• DIOL(Deep Intraoption learning) for high-level policy update
• DDPG(using double critics) + HER for low-level policy
• AAES(Auto-Adjusting Exploration Strategy) for stabilizing low-level policy in parallel training
• Abstract Demonstration to accelerate