Affordance Learning for End-to-End Visuomotor Robot Control
In Affordance Learning for End-to-End Visuomotor Robot Control, we introduced a modular deep neural network structure, that detects a container on a table, and inserts a ball into it.
We showed that our system performs its task successfully in zero-shot sim-to-real transfer manner. Each part of our system was fully trained either with synthetic data or in a simulation. The system was invariant to, e.g., distractor objects and textures.
The system structure:
We have didvided our work into the following code blocks:
- AffordanceVAED extracts affordance information from an observation image, and represents it as a latent space vector. Figure 1 shows the structure of the model.
- BlenderDomainRandomizer generates a domain randomized dataset for VAED.
- TrajectoryVAE represents trajectories in a low-dimensional latent space, and generates a trajectory based on a given latent vector.
- affordance_gym generates training data for TrajectoryVAE, and combines VAED and TrajectoryVAE together to perform desired trajectories based on an observation.
Ros Kinetic, MoveIt!, and MuJoco (2.0) should be installed. Colcon was used to build a workspace.
pip install -r affordance_gym/requirements.txt sudo apt install ros-kinetic-moveit ros-kinetic-libfranka ros-kinetic-franka-ros
The work environemnt depends also on the
mkdir -p ~/ros/src cd ~/ros/src git --recursive clone firstname.lastname@example.org:gamleksi/affordance_gym.git cd ~/ros colcon build
The folder structure:
- Lumi Testbed by Intelligent Robotics group: The set of core ROS packages for Lumi, our robot. Contains URDF description, moveit configuration, mujoco configuration.
- Mujoco Ros Control by Intelligent Robotics group: Interface for the MuJoCo simulator.
- Lumi Pose Estimation: computes the camera pose of Kinect with an aruco marker, which is required in kinect_simulation.launch.
- Affordance Gym: Affordance Learning for End-to-End Visuomotor Robot Control
Run (in the affordance_gym folder)
Training Flow of the System
- Generate a domain randomized dataset for affordance detection, train a VAED model, and update
VAED_MODELS_PATHin src/env_setup/env_setup.py to match to the model's parent folder path.
- Generate trajectory training data run
python scripts/generate_perception_data.py, train a TrajectoryVAE model, and update TRAJ_MODELS_PATH (in src/env_setup/env_setup.py) to match to the model's parent folder path.
POLICY_MODELS_PATH(in src/env_setup/env_setup.py) to match to the model's parent folder path and run
python scripts/generate_perception_data.pyto generate training policy data.
- To train a policy model run
python scripts/perception_policy_train.py, and to evaluate the policy model's performance in MuJoCo run
KINECT_EXPERIMENTS_PATH(in src/env_setup/env_setup.py) and run
python scripts/kinect_env.pyto experiment the learned model with a real camera and with or without a real robot.
More info about each phase run
python scripts/file_name.py -h.
Scripts and Roslaunch
The table shows which launch file is required to be running with each python file.
|generate_perception_data.py||roslaunch lumi_mujoco table_simulation.launch|
|generate_trajectories.py||roslaunch lumi_mujoco table_simulation.launch|
|kinect_env.py||roslaunch lumi_mujoco kinect_simulation.launch (without a real robot arm)|
|kinect_test.py||roslaunch lumi_mujoco kinect_simulation.launch|
|mc_interface.py||(real robot launch file)|
|perception_policy_eval.py||roslaunch lumi_mujoco table_simulation.launch|