SpaHybGen generates grasp poses for general robotic hands in SE(3) clutter scenes using learning and optimization strategies. It uses the real grasping dataset GraspNet-1Billion to train the inference section. SpaHybGen can directly and robustly deploy any robotic hand with its URDF to actual clutter grasping in minutes, using a single depth camera.
IMPORTANT: It is feasible to use your own robotic hands without Contact Dataset Generation and Contact Training. To immediately use your custom robotic hands, please directly refer to Pipeline: Grasp Generation (after setting the Code Environment).
✅ Autmatic objective-scaling strategy to grasp optimization, replacing constant hand-tuned scales.
✅ Replaced absolute path and magic number.
✅ Polished contact assignment script.
✅ Broke down functions and formated for clarity.
✅ Added functional documentation and type hints.
✅ Added pytest-based modular test [Code Test section]
We use Python 3.8 in Conda to train 3D U-Net, infer contacts, and optimize grasps. All algorithmic procedures are coded in Pytorch and Pytorch_kinematics. Environment setup for the real-world deployment refers to the following Actual Grasping section.
- Create virtual env.:
conda create --name spahybgen python=3.8
conda activate spahybgen- Install required packages with
pipon virtual env or python3:
pip install -r ./assets/requirements.txt
export SKLEARN_ALLOW_DEPRECATED_SKLEARN_PACKAGE_INSTALL=True # for sklearn package error
pip install graspnetAPI # for Graspnet Dataset- Install the project locally in "editable" mode using pip:
pip install -e .We understand that the setup of conda for GPU-based python packages can be tricky in varied machine and the above instruction may be insufficient for an error-free installation. Thus, we further share the specification of our installed env. in environment.yml for reference.
To validate the code and environment, it is recommanded to run modular tests:
pip install pytest
cd spahybgen
pytest tests/ -v
We release the generated contact dataset in Google Drive. It includes 4.5GB training data and 4.2GB test data.
If researchers expect to generate the contact dataset, please download the full GraspNet-1Billion dataset and run the following command:
cd spahybgen
python scripts/generate_dataset.py --graspnet PATH_TO_GraspNet --output ./dataset/trainIt will take tens of hours for the generation process (currently we have not parallelized it).
Detailed descriptions of the contact generation process are presented in scripts/generate_dataset.py.
The contact dataset should be placed inside a dataset folder as: spahybgen\dataset\train\scene_0000.
After generating or downloading the Contact Dataset in the previous step, run command to train a 3D U-Net:
python scripts/train_shgn.py --dataset dataset/train/ --net unet --orientation quat --gridtype voxel --batch-size 4 --numsample 3000 --epochs 64 --loaders 10 --gridtype voxelThe training logs and models are stored at data/runs/.
To facilitate reproduction, two trained models are also shared in Google Drive and assets/trained_models/, which contains two networks oriented to Voxel and TSDF input observations.
The process of grasp generation includes 1.scene observation, 2.contact inference, 3.hand model and 4.grasp optimization.
We enable two formats (Voxel and TSDF) as the scene observation in actual grasping tasks. To obtain observation, one can refer to the generated (downloaded) contact dataset in folder spahybgen/dataset/, where the .npz files are observations for grasping scenes in GraspNet-1Billion.
Alternatively, practitioners capture scene volumes using a depth sensor, following the sensing pipeline at src/spahybgen/pipeline/sensor_server.py.
Two observation samples can be found in assets/observations/. You can load them with
np.load('assets/observations/scene_010_ann_0124_voxel.npz')["grid"].
With the trained model and obtained observation, dense contact features can be reasoned before grasp optimization.
Note: If you want to individually test the contact inference module, please run:
python scripts/contact_inference_test.pyIt will infer contact features using the observation scene_010_ann_0124_voxel.npz and model spahybgen_unet_64_voxel.pt in folder assets/.
More than ten robotic hands are released in folder \handmodel. To construct a custom gripper in compatible format, please check these hand examples. Generally, one hand model can be generated within the following steps:
(1). Prepare the standard URDF file for the targeted robotic hand. The CAD filepath and xml encoding information in .urdf should be properly modified to match the code (for the targeted format, please refer to the released hand examples).
(2). Assign contact regions to the hand surface using the tool in scripts/hand_contacts.ipynb.
(3). Append information of the custom hand to file handmodel/hand_infos.json, following the included format.
With the inferred contact features and established hand model, grasp optimization is parallelized using Pytorch_kinematics.
After running the contact inference script contact_inference_test.py in Step 2, the following cmd will optimize grasps using the specific hand and visualize the results using Web-based Plotly:
python scripts/grasp_optimization_test.py --hand robotiq2f --max_iter 120 --batch_size 64
# AVAILABLE HAND MODELS:
# 2F: [robotiq2f, finray2f, antipodal_hand]
# 3F: [robotiq3f, softpneu3f]
# 4F: [finray4f, leaphand]
# 5F: [brunel_hand]Note: If you want to individually test the grasp optimization module, download the std_inference_result_from_clutter.npy from Google Drive to the folder ./assets, and rename it to inference_results.npy.
To run the full algorithmic pipeline without hardware (using the default observation file and trained model contained in folder assets/), please directly run:
python scripts/generation_pipeline_test.py --hand robotiq2f --max_iter 120 --batch_size 64This script is a combination of 2. Contact Inference and 4. Grasp Optimization. Similarly, it will optimize grasps using Robotiq-2F and visualize the results using Web-based Plotly.
-
In our actual grasping experiments, ROS Melodic is used to coordinate the UR-5e, Azure RGB-D camera, and multiple robotic hands. To facilitate a fast setup of the actual grasping framework, we released the ROS-based communication interfaces of all devices and algorithm modules, as in the folder src/spahybgen/pipeline/.
-
Practitioners are expected to set up the custom hardware with ROS and modify their specific ROS topics in different files (camera: config/grasp_generation.yaml; robot arm:src/spahybgen/pipeline/pose_node.py; robotic hands: src/spahybgen/pipeline/gripper_node.py).
-
Finally, scripts/realrobot_execution.py details a single-gripper grasping pipeline which includes hardware execution.
To enable ROS1 in Python3, please follow Coding_Instruction to make
import rospyandimport tf2_rosworking in python3.
@article{wang2026learning,
title={Learning contact representations in real-world clutter for universal robotic grasping},
author={Wang, Xianli and Tam, Lap Mou and Xu, Qingsong},
journal={Nature Machine Intelligence},
year={2026},
doi={https://doi.org/10.1038/s42256-026-01292-y},
publisher={Nature Publishing Group UK London}
}
This project is inspired by the excellent works VGN, GenDexGrasp, and GraspNetAPI.