RL MPC Locomotion

This repo is aim to provide a fast simulation and RL training platform for quadrupad locomotion. The control framework is a hierarchical controller composed of an higher-level policy network and a lower-level model predictive controller (MPC).

The MPC controller refers to Cheetah Software but written in python, and it completely opens the interface between sensor data and motor commands, so that the controller can be easily ported to mainstream simulators like MuJoCo.

The RL training is carried out in the NVDIA Isaac Gym in parallel using Unitree Robotics Aliengo model, and transferring it from simulation to reality on a real Aliengo robot.

Frameworks

Dependencies

• PyTorch - 1.8.1
• Isaac Gym - Preview 4
• RL Games - 1.1.3

Installation

1. Clone this repository

   git clone git@github.com:silvery107/rl-mpc-locomotion.git
   git submodule update --init

   Or use --recurse option to clone submodules at the same time.

2. Create the conda environment:

   conda env create -f environment.yml

3. Install the python binding of the MPC solver:

   pip install -e .

Quick Start

1. Play the MPC controller on Aliengo:

   python RL_MPC_Locomotion.py --robot=Aliengo

   All supported robot types are Aliengo, A1 and Mini_Cheetah. Note that by default you need to plug in your Xbox-like gamepad to control it.

   • Gamepad keymap

     Press LB to switch gait types between Trot, Fly Tort, Gallop, Walk and Pace.

     Press RB to switch FSM states between Locomotion and Recovery Stand

2. Train a new policy: Set bridge_MPC_to_RL to True in <MPC_Controller/Parameters.py>

   cd RL_Environment
   python train.py task=Aliengo headless=False

   Press the v key to disable viewer updates, and press again to resume. Set headless=True to train without rendering.

   Tensorboard support is avaliable, run tensorboard --logdir runs.

3. Load a trained checkpoint:

   python train.py task=Aliengo checkpoint=runs/Aliengo/nn/Aliengo.pth test=True num_envs=4

   Set test=False to continue training.

4. Run the trained weight-policy for MPC controller on Aliengo:

   python RL_MPC_Locomotion --robot=Aliengo --mode=Policy

   By default the controller mode is Fsm, and you can also try Min for the minimum MPC controller (without FSM).

Roadmaps

• MPC Controller
• Quadruped,
• RobotRunner ->
  • LegController,
  • StateEstimator,
  • ControlFSM ->
    • FSM State RecoveryStand,
    • FSM State Locomotion ->
      • ConvexMPCLocomotion ->
        • FootSwingTrajectory,
        • Gait,
        • MPC Solver in C

• RL Environment
• Gamepad Reader,
• Simulation Utils,
• Weight Policy,
• Train ->
  • Vectorized Env,
  • Aliengo Env

User Notes

• Setup a Simulation in Isaac Gym
• Install MIT Cheetah Software
• Upgrade Isaac Gym Preview 2 to Preview 3
• OSQP, qpOASES and CVXOPT Solver Instructions
• Development Logs

Gallery