Awesome Object-Centric Robotic Manipulation

This is the repository of A Survey of Embodied Learning for Object-Centric Robotic Manipulation, a comprehensive review of latest advancements in object-centric robotic manipulation including embodied perceptual learning, embodied policy learning, and embodied task-oriented learning. For details, please refer to:

A Survey of Embodied Learning for Object-Centric Robotic Manipulation

Any contributions to this list are most welcome! Please feel free to open issues or create pull requests as follows:

• Fork the project into your own repository.
• Add the Title, Paper link, Venue, Code/Project link in README.md using the following format:

  |[Title](Paper Link)|Venue|[Code/Project](Code/Project link)|
  

• Submit the pull request to this branch.

🔥 News

Last update on 2024/08/28

🔨 Abstract

Embodied learning for object-centric robotic manipulation is a rapidly developing and challenging area in embodied AI. It is crucial for advancing next-generation intelligent robots and has garnered significant interest recently. Unlike data-driven machine learning methods, embodied learning focuses on robot learning through physical interaction with the environment and perceptual feedback, making it especially suitable for robotic manipulation. In this paper, we provide a comprehensive survey of the latest advancements in this field and categorize the existing work into three main branches: 1) Embodied perceptual learning, which aims to predict object pose and affordance through various data representations; 2) Embodied policy learning, which focuses on generating optimal robotic decisions using methods such as reinforcement learning and imitation learning; 3) Embodied task-oriented learning, designed to optimize the robot's performance based on the characteristics of different tasks in object grasping and manipulation. In addition, we offer an overview and discussion of public datasets, evaluation metrics, representative applications, current challenges, and potential future research directions.

📝 Citation

@article{zheng2024ocrm,
  author= {Zheng, Ying and Yao, Lei and Su, Yuejiao and Zhang, Yi and Wang, Yi and Zhao, Sicheng and Zhang, Yiyi and Chau, Lap-Pui},
  title= {A Survey of Embodied Learning for Object-Centric Robotic Manipulation},
  journal={arXiv preprint arXiv:2408.11537},
  year={2024}
}

• Awesome Object-Centric Robotic Manipulation
  • 🔥 News
  • 🔨 Abstract
  • 📝 Citation
  • Embodied Perceptual Learning
    • Data Representation
      • Image-Based Representation
      • 3D-Aware Representation
      • Tactile-Based Representation
    • Object Pose Estimation
      • Instance-Level Object Pose Estimation
      • Category-Level Object Pose Estimation
      • Novel Object Pose Estimation
    • Affordance Learning
      • Affordance Learning by Supervised Learning
      • Affordance Learning From Interaction
  • Embodied Policy Learning
    • Policy Representation
      • Explicit Policy
      • Implicit Policy
      • Diffusion Policy
    • Policy Learning
      • Reinforcement Learning
      • Imitation Learning
      • Other Methods
  • Embodied Task-Oriented Learning
    • Object Grasping
      • Single-Object Grasping
      • Multi-Object Grasping
    • Object Manipulation
      • Non-Dexterous Manipulation
      • Dexterous Manipulation
  • Datasets
    • Datasets for Object Grasping
    • Datasets for Object Manipulation
  • Applications
    • Industrial Robots
    • Agricultural Robots
    • Domestic Robots
    • Surgical Robots
    • Other Applications
  • Challenges and Future Directions
    • Sim-to-Real Generalization
    • Multimodal Embodied LLMs
    • Human-Robot Collaboration
    • Model Compression and Robot Acceleration
    • Model Interpretability and Application Safety
  • 📚 License

Embodied Perceptual Learning

Data Representation

Image-Based Representation

Paper	Venue	Code/Project
Real-time seamless single shot 6d object pose prediction	CVPR 2018	-
MonoGraspNet: 6-DoF Grasping with a Single RGB Image	ICRA 2023	-
RGBGrasp: Image-based Object Grasping by Capturing Multiple Views during Robot Arm Movement with Neural Radiance Fields	RAL 2024	Project
RGBManip: Monocular Image-based Robotic Manipulation through Active Object Pose Estimation	ICRA 2024	Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects	CoRL 2022	Project
Clear-Splatting: Learning Residual Gaussian Splats for Transparent Object Manipulation	ICRAW 2024	-

3D-Aware Representation

Depth-based representations

Paper	Venue	Code/Project
RGB-D object detection and semantic segmentation for autonomous manipulation in clutter	IJRR 2018	-

Point cloud-based representations

Paper	Venue	Code/Project
Shape completion enabled robotic grasping	IROS 2017	-
PointNetGPD: Detecting Grasp Configurations from Point Sets	ICRA 2019	Project
3D Implicit Transporter for Temporally Consistent Keypoint Discovery	ICCV 2023	Code

Other representations

Paper	Venue	Code/Project
RVT: Robotic View Transformer for 3D Object Manipulation	CoRL 2023	Project
GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fields	CoRL 2023	Project
ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation	ECCV 2024	Project
Object-Aware Gaussian Splatting for Robotic Manipulation	ICRAW 2024	Project

Tactile-Based Representation

Paper	Venue	Code/Project
GelSight: High-Resolution Robot Tactile Sensors for Estimating Geometry and Force	Sensors 2017	-
Digit: A novel design for a low-cost compact high-resolution tactile sensor with application to in-hand manipulation	RAL 2020	Project
Allsight: A low-cost and high-resolution round tactile sensor with zero-shot learning capability	RAL 2023	Code
Learning to grasp without seeing	ISER 2018	Project
Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Tasks	TRO 2020	Project
The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?	CoRL 2017	-
Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Play	CoRL 2023	Project
More Than a Feeling: Learning to Grasp and Regrasp using Vision and Touch	RAL 2018	-
ObjectFolder: A Dataset of Objects with Implicit Visual, Auditory, and Tactile Representations	CoRL 2022	Project
TACTO: A Fast, Flexible, and Open-source Simulator for High-Resolution Vision-based Tactile Sensors	CoRL 2022	Code
Efficient Tactile Simulation with Differentiability for Robotic Manipulation	CoRL 2023	Project
Touching a NeRF: Leveraging Neural Radiance Fields for Tactile Sensory Data Generation	CoRL 2023	-
Tactile-Augmented Radiance Fields	CVPR 2024	Project

Object Pose Estimation

Instance-Level Object Pose Estimation

Paper	Venue	Code/Project
PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes Introduction	RSS 2018	Project
CDPN: Coordinates-Based Disentangled Pose Network for Real-Time RGB-Based 6-DoF Object Pose Estimation	ICCV 2019	-
6D-Diff: A Keypoint Diffusion Framework for 6D Object Pose Estimation	CVPR 2024	-
Match Normalization: Learning-Based Point Cloud Registration for 6D Object Pose Estimation in the Real World	TPAMI 2024	-
Deep fusion transformer network with weighted vector-wise keypoints voting for robust 6d object pose estimation	ICCV 2023	Code

Category-Level Object Pose Estimation

Paper	Venue	Code/Project
Normalized object coordinate space for category-level 6d object pose and size estimation	CVPR 2019	Project
Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation	ICCV 2021	Project
Rigidity-aware detection for 6d object pose estimation	CVPR 2023	-
Revisiting Fully Convolutional Geometric Features for Object 6D Pose Estimation	ICCV 2023	Code
Category-Level Articulated Object Pose Estimation	CVPR 2020	Project
Towards Real-World Category-level Articulation Pose Estimation	TIP 2022	-
CenterSnap: Single-Shot Multi-Object 3D Shape Reconstruction and Categorical 6D Pose and Size Estimation	ICRA 2022	Project
ShAPO: Implicit Representations for Multi Object Shape Appearance and Pose Optimization	ECCV 2022	Project
FSD: Fast Self-Supervised Single RGB-D to Categorical 3D Objects	ICRA 2024	Project
DiffusionNOCS: Managing Symmetry and Uncertainty in Sim2Real Multi-Modal Category-level Pose Estimation	IROS 2024	Project

Novel Object Pose Estimation

Paper	Venue	Code/Project
Gen6D: Generalizable Model-Free 6-DoF Object Pose Estimation from RGB Images	ECCV 2022	Project
GigaPose: Fast and Robust Novel Object Pose Estimation via One Correspondence	CVPR 2024	Code
OnePose: One-Shot Object Pose Estimation without CAD Models	CVPR 2022	Project
Zero-Shot Category-Level Object Pose Estimation	ECCV 2022	Code
SAM-6D: Segment Anything Model Meets Zero-Shot 6D Object Pose Estimation	CVPR 2024	Code
FoundationPose: Unified 6D Pose Estimation and Tracking of Novel Objects	CVPR 2024	Project

Affordance Learning

Affordance Learning by Supervised Learning

Paper	Venue	Code/Project
Affordancenet: An end-to-end deep learning approach for object affordance detection	ICRA 2018	Code
Grounded Human-Object Interaction Hotspots From Video	ICCV 2019	Porject
Affordances from Human Videos as a Versatile Representation for Robotics	CVPR 2023	Project
Robo-ABC: Affordance Generalization Beyond Categories via Semantic Correspondence for Robot Manipulation	ECCV 2024	Porject
RAM: Retrieval-Based Affordance Transfer for Generalizable Zero-Shot Robotic Manipulation	arXiv 2024	Project
Open-Vocabulary Affordance Detection in 3D Point Clouds	IROS 2023	Project
Learning 2D Invariant Affordance Knowledge for 3D Affordance Grounding	arXiv 2024	Code

Affordance Learning From Interaction

Paper	Venue	Code/Project
Where2Act: From Pixels to Actions for Articulated 3D Objects	ICCV 2021	Project
AdaAfford: Learning to Adapt Manipulation Affordance for 3D Articulated Objects via Few-shot Interactions	ECCV 2022	Porject
DualAfford: Learning Collaborative Visual Affordance for Dual-gripper Manipulation	ICLR 2023	Project
Self-Supervised Learning of Action Affordances as Interaction Modes	ICRA 2023	Porject
Information-driven Affordance Discovery for Efficient Robotic Manipulation	arXiv 2024	Project
Where2Explore: Few-shot Affordance Learning for Unseen Novel Categories of Articulated Objects	NeurIPS 2023	Project
Articulated Object Manipulation with Coarse-to-fine Affordance for Mitigating the Effect of Point Cloud Noise	ICRA 2024	-
Learning environment-aware affordance for 3d articulated object manipulation under occlusions	NeurIPS 2023	Code
End-to-End Affordance Learning for Robotic Manipulation	ICRA 2023	Project

Embodied Policy Learning

Policy Representation

Explicit Policy

Paper	Venue	Code/Project
Robotic grasping using deep reinforcement learning	CASE 2020	-
Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulation	CoRL 2018	Porject
Rapid Motor Adaptation for Robotic Manipulator Arms	CVPR 2024	-
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot	IROS 2023	Code
Learning insertion primitives with discrete-continuous hybrid action space for robotic assembly tasks	ICRA 2022	Project

Implicit Policy

Paper	Venue	Code/Project
Regularizing Model-Based Planning with Energy-Based Models	CoRL 2019	-
Implicit Behavioral Cloning	CoRL 2021	-
Implicit Distributional Reinforcement Learning	NeurIPS 2020	Code
Energy-Based Imitation Learning	AAMAS 2021	Code

Diffusion Policy

Paper	Venue	Code/Project
Diffusion policies as an expressive policy class for offline reinforcement learning	ICLR 2023	Code
Diffusion Policy: Visuomotor Policy Learning via Action Diffusion	RSS 2023	Project
Is conditional generative modeling all you need for decision-making?	ICLR 2023	-
Hierarchical Diffusion Policy for Kinematics-Aware Multi-Task Robotic Manipulation	CVPR 2024	Project
Dexterous Functional Pre-Grasp Manipulation with Diffusion Policy	arXiv 2024	Project
Multimodal Diffusion Transformer: Learning Versatile Behavior from Multimodal Goals	ICRAW 2024	-
Language-Guided Object-Centric Diffusion Policy for Collision-Aware Robotic Manipulation	arXiv 2024	-
Goal-Conditioned Imitation Learning using Score-based Diffusion Policies	RSS 2023	Project

Policy Learning

Reinforcement Learning

Paper	Venue	Code/Project
Learning dexterous in-hand manipulation	IJRR 2020	-
Value-Informed Skill Chaining for Policy Learning of Long-Horizon Tasks with Surgical Robot	IROS 2023	Code
Rapid Motor Adaptation for Robotic Manipulator Arms	CVPR 2024	-
SAM-RL: Sensing-Aware Model-Based Reinforcement Learning via Differentiable Physics-Based Simulation and Rendering	RSS 2023	Project
What Matters in Learning from Offline Human Demonstrations for Robot Manipulation	CoRL 2021	Project
Demonstration-Guided reinforcement learning with efficient exploration for task automation of surgical robot	ICRA 2023	Code
Text2Reward: Reward Shaping with Language Models for Reinforcement Learning	ICLR 2024	Project
Eureka: Human-Level Reward Design via Coding Large Language Models	ICLR 2024	Project

Imitation Learning

Paper	Venue	Code/Project
Behavioral Cloning from Observation	IJCAI 2018	-
Maximum Entropy Inverse Reinforcement Learning	AAAI 2008	-
SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networks	ICRA 2024	Project
Transformer-based deep imitation learning for dual-arm robot manipulation	IROS 2021	Project
Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware	RSS 2023	Project
MimicGen: A Data Generation System for Scalable Robot Learning using Human Demonstrations	CoRL 2023	Project
Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasets	RSS 2022	Project
Vid2Robot: End-to-end Video-conditioned Policy Learning with Cross-Attention Transformers	RSS 2024	Project
Ag2Manip: Learning Novel Manipulation Skills with Agent-Agnostic Visual and Action Representations	ICRA 2024	Project
Learning Manipulation by Predicting Interaction	RSS 2024	Project
Neural Descriptor Fields: SE(3)-Equivariant Object Representations for Manipulation	ICRA 2022	Project
Local Neural Descriptor Fields: Locally Conditioned Object Representations for Manipulation	ICRA 2023	Project
Equivariant Descriptor Fields: SE(3)-Equivariant Energy-Based Models for End-to-End Visual Robotic Manipulation Learning	ICLR 2023	Code
Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation	CVPR 2024	Project
EDGI: Equivariant Diffusion for Planning with Embodied Agents	NeurIPS 2023	-
SE(3)-DiffusionFields: Learning smooth cost functions for joint grasp and motion optimization through diffusion	ICRA 2023	Project

Other Methods

Paper	Venue	Code/Project
UniDexGrasp: Universal Robotic Dexterous Grasping via Learning Diverse Proposal Generation and Goal-Conditioned Policy	CVPR 2023	Project
UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learning	ICCV 2023	-
Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planning	arXiv 2023	Project
Grounding Multimodal Large Language Models in Actions	arXiv 2024	-
OpenVLA: An Open-Source Vision-Language-Action Model	arXiv 2024	Project
3D-VLA: A 3D Vision-Language-Action Generative World Model	ICML 2024	Code
RoboMP2: A Robotic Multimodal Perception-Planning Framework with Multimodal Large Language Models	ICML 2024	Project

Embodied Task-Oriented Learning

Object Grasping

Single-Object Grasping

Paper	Venue	Code/Project
Deep Learning for Detecting Robotic Grasps	IJRR 2015	-
Real-Time Grasp Detection Using Convolutional Neural Networks	ICRA 2015	-
RGB-D Object Recognition and Grasp Detection using Hierarchical Cascaded Forests	TRO 2017	-
FANet: Fast and Accurate Robotic Grasp Detection Based on Keypoints	TASE 2023	-
Anygrasp: Robust and efficient grasp perception in spatial and temporal domains	TRO 2023	-
Dynamic grasp and trajectory planning for moving objects	Autonomous Robots 2019	-
Closing the Loop for Robotic Grasping: A Real-time, Generative Grasp Synthesis Approach	RSS 2018	Code
DITTO: Demonstration Imitation by Trajectory Transformation	Autonomous Robots 2019	Project
D-Grasp: Physically Plausible Dynamic Grasp Synthesis for Hand-Object Interactions	CVPR 2022	Project
GraspXL: Generating Grasping Motions for Diverse Objects at Scale	ECCV 2024	Project

Transparent Object Grasping.

Paper	Venue	Code/Project
Transcg: A large-scale real-world dataset for transparent object depth completion and a grasping baseline	RAL 2022	Project
Dex-NeRF: Using a Neural Radiance Field to Grasp Transparent Objects	CoRL 2021	Project
Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects	CoRL 2022	Project
Graspnerf: Multiview-based 6-dof grasp detection for transparent and specular objects using generalizable nerf	ICRA 2023	Project
NFL: Normal Field Learning for 6-DoF Grasping of Transparent Objects	RAL 2023	Project
TRansPose: Large-scale multispectral dataset for transparent object	IJRR 2024	Project
TODE-Trans: Transparent Object Depth Estimation with Transformer	ICRA 2023	Code

Grasping in Clutter.

Paper	Venue	Code/Project
Contact-GraspNet: Efficient 6-DoF Grasp Generation in Cluttered Scenes	ICRA 2021	Project
CaTGrasp: Learning Category-Level Task-Relevant Grasping in Clutter from Simulation	ICRA 2022	Project
6-DOF Grasping for Target-driven Object Manipulation in Clutter	ICRA 2020	-
DDGC: Generative Deep Dexterous Grasping in Clutter	RAL 2021	Project
Graspness Discovery in Clutters for Fast and Accurate Grasp Detection	ICCV 2021	-
Discriminative Active Learning for Robotic Grasping in Cluttered Scene	RAL 2023	-
Efficient learning of goal-oriented push-grasping synergy in clutter	RAL 2021	Code
Robust object grasping in clutter via singulation	ICRA 2019	-
A deep learning approach to grasping the invisible	RAL 2020	Project
A joint modeling of vision-language-action for target-oriented grasping in clutter	ICRA 2023	Code

Dynamic Object Grasping.

Paper	Venue	Code/Project
Predicting Human Intentions in Human–Robot Hand-Over Tasks Through Multimodal Learning	TASE 2021	-
Reactive Human-to-Robot Handovers of Arbitrary Objects	ICRA 2021	Project
Flexible Handover with Real-Time Robust Dynamic Grasp Trajectory Generation	IROS 2023	-
Learning Human-to-Robot Handovers from Point Clouds	CVPR 2023	Project
GenH2R: Learning Generalizable Human-to-Robot Handover via Scalable Simulation Demonstration and Imitation	CVPR 2024	Project
SynH2R: Synthesizing Hand-Object Motions for Learning Human-to-Robot Handovers	ICRA 2024	Project
Dynamic Grasping with Reachability and Motion Awareness	IROS 2021	Project
Target-referenced Reactive Grasping for Dynamic Objects	CVPR 2023	Project
ArtiGrasp: Physically Plausible Synthesis of Bi-Manual Dexterous Grasping and Articulation	3DV 2024	Project

Multi-Object Grasping

Paper	Venue	Code/Project
Multi-Object Grasping in the Plane	ISRR 2022	Project
Multi-Object Grasping - Types and Taxonomy	ICRA 2022	-
Learning to Efficiently Plan Robust Frictional Multi-Object Grasps	IROS 2023	-
Multi-Object Grasping—Experience Forest for Robotic Finger Movement Strategies	RAL 2024	-
Push-MOG: Efficient Pushing to Consolidate Polygonal Objects for Multi-Object Grasping	CASE 2023	Project
Exploiting Kinematic Redundancy for Robotic Grasping of Multiple Objects	TRO 2023	-
Grasp multiple objects with one hand	RAL 2024	Project

Object Manipulation

Non-Dexterous Manipulation

Paper	Venue	Code/Project
Pick and place without geometric object models	ICRA 2018	-
Self-supervised Learning for Precise Pick-and-place without Object Model	RAL 2020	Project
SG-Bot: Object Rearrangement via Coarse-to-Fine Robotic Imagination on Scene Graphs	ICRA 2024	Project
Form2fit: Learning shape priors for generalizable assembly from disassembly	ICRA 2020	Project
Mesh-based Dynamics with Occlusion Reasoning for Cloth Manipulation	RSS 2022	Project
Learning where to trust unreliable models in an unstructured world for deformable object manipulation	Science Robotics 2021	-
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts	CVPR 2023	Project
UniDoorManip: Learning Universal Door Manipulation Policy Over Large-scale and Diverse Door Manipulation Environments	arXiv 2024	Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations	CVPR 2023	Project
Look Closer: Bridging Egocentric and Third-Person Views With Transformers for Robotic Manipulation	RAL 2022	Project
Self-Corrected Multimodal Large Language Model for End-to-End Robot Manipulation	arXiv 2024	Project

Dexterous Manipulation

Paper	Venue	Code/Project
DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation	CoRL 2023	Project
MyoDex: A Generalizable Prior for Dexterous Manipulation	ICML 2023	Project
Deep Dynamics Models for Learning Dexterous Manipulation	CoRL 2020	Project
Dexterous Imitation Made Easy: A Learning-Based Framework for Efficient Dexterous Manipulation	ICRA 2023	Project
Visual Dexterity: In-hand Dexterous Manipulation from Depth	ICMLW 2023	-
Dexdeform: Dexterous deformable object manipulation with human demonstrations and differentiable physics	ICLR 2023	Project

Tool Manipulation.

Paper	Venue	Code/Project
Tactile Tool Manipulation	ICRA 2023	-
Understanding Tools: Task-Oriented Object Modeling, Learning and Recognition	CVPR 2015	-
How to select and use tools? : Active Perception of Target Objects Using Multimodal Deep Learning	RAL 2021	-
KETO: Learning Keypoint Representations for Tool Manipulation	ICRA 2020	-
Learning Task-Oriented Grasping for Tool Manipulation from Simulated Self-Supervision	IJRR 2020	Project
DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools	CoRL 2023	Project
Leveraging Language for Accelerated Learning of Tool Manipulation	ICLR 2022	Project
Creative robot tool use with large language models	arXiv 2023	Project

Datasets

Datasets for Object Grasping

Paper	Name	Venue	Code/Project
Efficient grasping from rgbd images: Learning using a new rectangle representation	Cornell	ICRA 2011	-
Real-World Multiobject, Multigrasp Detection	Multi-Object	RAL 2018	Code
Jacquard: A Large Scale Dataset for Robotic Grasp Detection	Jacquard	IROS 2018	Project
Learning 6-DOF Grasping Interaction via Deep Geometry-aware 3D Representations	VR-Grasping-101	ICRA 2018	Project
ACRONYM: A Large-Scale Grasp Dataset Based on Simulation	ACRONYM	ICRA 2021	Project
Egad! an evolved grasping analysis dataset for diversity and reproducibility in robotic manipulation	EGAD	RAL 2020	Project
Graspnet-1billion: A large-scale benchmark for general object grasping	GraspNet-1Billion	CVPR 2020	Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation Models	Grasp-Anything	ICRA 2024	Project

Datasets for Object Manipulation

Paper	Name	Venue	Code/Project
Yale-CMU-Berkeley dataset for robotic manipulation research	YCB	IJRR 2017	Project
AKB-48: A Real-World Articulated Object Knowledge Base	Multi-Object	CVPR 2022	Project
SAPIEN: A SimulAted Part-based Interactive ENvironment	PartNet-Mobility	CVPR 2020	Project
GAPartNet: Cross-Category Domain-Generalizable Object Perception and Manipulation via Generalizable and Actionable Parts	GAPartNet	CVPR 2023	Project
Maniskill2: A unified benchmark for generalizable manipulation skills	ManiSkill2	ICLR 2023	Project
ARNOLD: A Benchmark for Language-Grounded Task Learning With Continuous States in Realistic 3D Scenes	ARNOLD	ICCV 2023	Project
Bi-DexHands: Towards Human-Level Bimanual Dexterous Manipulation	Bi-DexHands	TPAMI 2023	Code
DexArt: Benchmarking Generalizable Dexterous Manipulation with Articulated Objects	DexArt	CVPR 2023	Project
Grasp-Anything: Large-scale Grasp Dataset from Foundation Models	Grasp-Anything	ICRA 2024	Project
PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observations	PartManip	CVPR 2023	Project
BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation	BEHAVIOR-1K	CoRL 2022	Project

Applications

Industrial Robots

Paper	Venue	Code/Project
Robot learning of industrial assembly task via human demonstrations	Autonomous Robots 2019	-
Applying a 6 DoF robotic arm and digital twin to automate fan-blade reconditioning for aerospace maintenance, repair, and overhaul	Sensors 2020	-
Advanced predictive maintenance with machine learning failure estimation in industrial packaging robots	DAS 2020	-
Compound fault diagnosis for industrial robots based on dual-transformer networks	Manu. Sys. 2023	-

Agricultural Robots

Paper	Venue	Code/Project
Technological revolutions in smart farming: Current trends, challenges & future directions	Comput Electron Agr 2022	-
Robotics in agriculture: Advanced technologies in livestock farming and crop cultivation	E3S Web Conf. 2024	-
State-of-the-art robotic grippers, grasping and control strategies, as well as their applications in agricultural robots: A review	Comput Electron Agr 2020	-
Current status and future challenges in implementing and upscaling vertical farming systems	Nature Food 2021	-
Intelligent robots for fruit harvesting: Recent developments and future challenges	Precis Agric 2022	-

Domestic Robots

Paper	Venue	Code/Project
Learning arbitrary-goal fabric folding with one hour of real robot experience	CoRL 2021	Project
RCareWorld: A Human-centric Simulation World for Caregiving Robots	IROS 2022	Project
Improving Robotic Cooking Using Batch Bayesian Optimization	RAL 2022	-
Robot Cooking With Stir-Fry: Bimanual Non-Prehensile Manipulation of Semi-Fluid Objects	RAL 2022	-
Efficient Task/Motion Planning for a Dual-arm Robot from Language Instructions and Cooking Images	IROS 2022	-
Continuous Object State Recognition for Cooking Robots Using Pre-Trained Vision-Language Models and Black-box Optimization	RAL 2024	-

Surgical Robots

Paper	Venue	Code/Project
Autonomous tissue manipulation via surgical robot using learning based model predictive control	ICRA 2019	Project
Super deep: A surgical perception framework for robotic tissue manipulation using deep learning for feature extraction	ICRA 2021	-
Transition state clustering: Unsupervised surgical trajectory segmentation for robot learning	IJRR 2017	Project
SurRoL: An Open-source Reinforcement Learning Centered and dVRK Compatible Platform for Surgical Robot Learning	IROS 2021	Code
Towards Human-Robot Collaborative Surgery: Trajectory and Strategy Learning in Bimanual Peg Transfer	RAL 2023	-
Human-in-the-loop Embodied Intelligence with Interactive Simulation Environment for Surgical Robot Learning	RAL 2023	Project
AutoLaparo: A New Dataset of Integrated Multi-tasks for Image-guided Surgical Automation in Laparoscopic Hysterectomy	MICCAI 2022	Project
ORBIT-Surgical: An Open-Simulation Framework for Learning Surgical Augmented Dexterity	ICRA 2024	Project

Other Applications

Paper	Venue	Code/Project
Reinforcement learning in dual-arm trajectory planning for a free-floating space robot	Aerosp Sci Technol 2020	Project
Robust Adaptive Learning Control of Space Robot for Target Capturing Using Neural Network	TNNLS 2023	-
RLBench: The Robot Learning Benchmark & Learning Environment	RAL 2020	Project

Challenges and Future Directions

Sim-to-Real Generalization

Paper	Venue	Code/Project
Using Simulation and Domain Adaptation to Improve Efficiency of Deep Robotic Grasping	ICRA 2018	Project
Sim-to-Real Transfer of Robotic Control with Dynamics Randomization	ICRA 2018	-
Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World	IROS 2017	Project
On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer	RAL 2024	-
TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction	arXiv 2024	Project
Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulation	CVPR 2024	Project
Generalizing 6-DoF Grasp Detection via Domain Prior Knowledge	RSS 2024	Project
Sim-to-Real 6D Object Pose Estimation via Iterative Self-training for Robotic Bin Picking	ECCV 2022	Project
Kalman Filter-Based One-Shot Sim-to-Real Transfer Learning	RAL 2023	-

Multimodal Embodied LLMs

Paper	Venue	Code/Project
Embodied Task Planning with Large Language Models	arXiv 2023	Project
PaLM-E: An Embodied Multimodal Language Model	ICML 2023	Project
ManipLLM: Embodied Multimodal Large Language Model for Object-Centric Robotic Manipulation	CVPR 2024	Project
Reasoning Tuning Grasp: Adapting Multi-Modal Large Language Models for Robotic Grasping	CoRLW 2023	Project
ManipVQA: Injecting Robotic Affordance and Physically Grounded Information into Multi-Modal Large Language Models	IROS 2024	Code
A3VLM: Actionable Articulation-Aware Vision Language Mode	arxiv 2024	Code

Human-Robot Collaboration

Paper	Venue	Code/Project
Progress and Prospects of the Human-Robot Collaboration	Autonomous Robots 2018	-
A Learning Based Hierarchical Control Framework for Human–Robot Collaboration	TASE 2022	-
Co-GAIL: Learning Diverse Strategies for Human-Robot Collaboration	CoRL 2022	Project

Model Compression and Robot Acceleration

Paper	Venue	Code/Project
RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control	CoRL 2023	Project

Model Interpretability and Application Safety

Paper	Venue	Code/Project
Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation	IROS 2019	Project
Efficient and Interpretable Robot Manipulation with Graph Neural Networks	RAL 2022	-
Interpretable Robotic Manipulation from Language	arXiv 2024	Code
Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmark	NeurIPS 2023	Project
Physical Adversarial Attack on a Robotic Arm	RAL 2022	-
SMOF - A Safety MOnitoring Framework for Autonomous Systems	TSMC 2016	-

📚 License

This repository is released under the MIT license.