MultiGripperGrasp Toolkit

Simulation based grasp filter. This repository contains a grasp filter developed using Isaac Sim, it provides the usage and visualization tools for the MultiGripperGrasp dataset.

MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Hands

Luis Felipe Casas Murrilo, Ninad Khargonkar, Balakrishnan Prabhakaran, Yu Xiang

[paper] [video] [arXiv] [project site] [dataset folder]

Citing MulitGripperGrasp Toolkit

Please cite MulitGripperGrasp Toolkit if it helps your research:

@misc{murrilo2024multigrippergrasp,
  title={MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Hands}, 
  author={Luis Felipe Casas Murrilo and Ninad Khargonkar and Balakrishnan Prabhakaran and Yu Xiang},
  year={2024},
  eprint={2403.09841},
  archivePrefix={arXiv},
  primaryClass={cs.RO}
}

License

MultiGripperGrasp Toolkit is released under the GNU General Public License v3.0.

Contents

• MultiGripperGrasp Toolkit
  • Citing MulitGripperGrasp Toolkit
  • License
  • Contents
  • Prerequisites
  • Installation
    • Alternate Visualization Tool
  • Running the Simulation
    • Important Files/Folders description
    • Parameters and Inputs
    • Transferred Grasps
    • Reproducing Paper Results
  • More Documentation
    • Helpful Links
    • Notes

Prerequisites

This code was tested with Isaac Sim 2023.1.0 on Linux.

Installation

1. Clone the repo:

   git clone git@github.com:IRVLUTD/isaac_sim_grasping.git

2. Install the Isaac Sim 2023.1.0 and cd to its python directory: Follow the instructions in the Isaac Sim Website for proper installation. Normally the python.sh is on the following directory:

   cd ~/.local/share/ov/pkg/isaac_sim-2023.1.0

3. Install the required libraries using the Isaac Sim python environment:

    ./python.sh -m pip install tqdm

   Note: Only tqdm must be installed, all other libraries should come with the Isaac Sim version.

4. Download the MultiGripperGrasp dataset and the object .usd models from the dataset folder. Links can also be found at the end of this document.

Visualizing Dataset Samples

To visualize the dataset a visualization standalone is included in the repository. The script must be ran with the Isaac Sim python.sh. The script requires the following parameters:

• json_dir: Grasp data directory (dataset .json folder). Note: the program does not read .json files within folders, therefore, a valid directory would be the Allegro folder within the dataset.
• gripper_dir: Gripper directory (folder containing all the gripper .usd folders)
• objects_dir: Object directory (folder containing all the object .usd folders)
• num_w: Number of Workstations to run simultaneously (gripper, object pair) (default: 15)
• (Optional) ub: Upper bound to filter the grasps with (applied to fall time).
• (Optional) lb: Lower bound to filter the grasps with (applied to fall time).
• (Optional) transfer: Used for visualizing transfered grasps.
• (Optional) device: Specifies which gpu device to run Isaac Sim with.

Note: To run the simulation without warnings add the following parameters to the command:

--/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Example command to run the visualization script:

./python.sh (repo directory)/visualize_grasps.py --json_dir=(dataset .json folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --num_w=10 --ub=3 --lb=0 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Alternate Visualization Tool

Please refer to the scripts under the viz_optas folder for an alternative way to visualize the grasps without the IsaacSim dependency!

Running the Simulation

The simulation can use any gripper and object provided they are prepared correctly (See the documentation to add grippers and objects). It loads the grasp information from the files specified and creates multiple "workstations" to test all the grasps. Then, it tries to perform the grasps with the specified control routines. When the object falls or the testing time is up, the time they took to fall is recorded and then saved to the output file. If any grasps is unable to touch the object, it will be recorded as a negative "fall time" value. Once the grasp test is completed, the workstation will reset and load a new grasp for testing. When all the grasps are finished, the output file is saved, the complete simulation will reset and a new file loaded.

Important Files/Folders description

1. standalone.py: Graspit grasps standalone executable
2. transfer_st.py: Transfered grasps standalone executable
3. visualize_grasps.py: Standalone executable to visualize filtered grasps
4. views.py: Simulation's behavioral code
5. managers.py: Contains grasp information and the reporting of results for all the standalones
6. controllers.py: Developed gripper controllers
7. utils.py: General utility functions
8. grippers: Gripper .usd files

Parameters and Inputs

A standalone executable for the graspit grasps (standalone.py) and for transferred grasps (transfer_st.py) are included within the repository. A sample command to run the graspit grasps is shown below along with the available options. Note: for Isaac Sim standalone executables, the commands must be run from the isaac sim python.sh directory.

The standalone.py takes as input:

• json_dir: Grasp data to filter directory (dataset .json folder). Note: the program does not read .json files within folders, therefore, a valid directory would be the Allegro folder within the dataset.
• gripper_dir: Gripper directory (folder containing all the gripper .usd folder)
• objects_dir: Object directory (folder containing all the object .usd folder)
• output_dir: Output directory (directory to save the filtered grasps .json file)
• num_w: Number of Workstations to run simultaneously (object-gripper pair) (default: 150)
• test_time: Total test time for each grasp test (default:3).
• controller: Controller reference (within controllers.py)(default: position)
• (Optional) print_results: Verbosity of standalone after finishing one .json file.
• (Optional) device: GPU device to use for Isaac Sim (default: 0)
• (Optional) headless: Run the simulation headless

Note: To run the simulation without warnings add the following parameters to the command:

 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Sample run command:

./python.sh (repo directory)/standalone.py --json_dir=(.json files folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --output_dir=(output directory) --num_w=100 --test_time=3 --controller=position --headless --print_results --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Transferred Grasps

The grasp filter is able to evaluate the object fall-off time for a large amount of generated grasps. The successful grasps of one gripper can represent successful grasps in others and increase the overall amount of successful grasps in the dataset. To test this hypothesis, we implemented the grasp transfer of successful grasps from one gripper to others and evaluated the transferred grasps using antoher Isaac Sim standalone (transfer_st.py).

We utilize an alignment between grippers to transfer grasps, which used a common notion of gripper pose (translation and orientation). The translation refers to the palm center of the gripper, and the orientation is with respect to a canonical pose of the gripper palm pointing in a fixed direction. Thus, any grasp pose from a gripper was transferred to another by using this pose alignment. We first transform a source gripper pose to its aligned pose, and then transform the aligned pose to the target gripper.

The transfer_st.py takes as input:

• json_dir: Grasp data to filter directory (dataset .json file folder). Note: the program does not read .json files within folders, therefore, a valid directory would be the Allegro folder within the dataset.
• gripper_dir: Gripper directory (folder containing all the gripper .usd folders)
• objects_dir: Object directory (folder containing all the object .usd folders)
• output_dir: Output directory (directory to save the filtered grasps .json file)
• num_w: Number of Workstations to run simultaneously (object-gripper pair) (default: 150)
• test_time: Total test time for each grasp test (default:3).
• controller: Controller reference (within controllers.py) (default: transfer_default)
• (Optional) print_results: Verbosity of standalone after finishing one .json file.
• (Optional) device: GPU device to use for Isaac Sim (default: 0)
• (Optional) headless: Run the simulation headless

Note: To run the simulation without warnings add the following parameters to the command:

 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Sample run command:

./python.sh (repo directory)/transfer_st.py --json_dir=(.json files folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --output_dir=(output directory) --num_w=100 --test_time=3 --headless --print_results --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Reproducing Paper Results

As of this version, the simulation uses a specific .json structure. The managers.py file is the responsible for reading and transforming the relevant grasp information to the format used by the program. The information is:

• The gripper name: Must be exactly the name with which the gripper is saved within the gripper directory.
• The object name: Must be exactly the name with which the object is saved within the object directory.
• Grasp pose data: 7 dimensional vector containing the gripper and object relative pose. (0-2 denotes the postion, 3-6 denotes the quaternion). Must be changed to the Isaac Sim quaternion convention (w, x, y, z)
• Joint Value Data: vector containing the Degree of Freedom (DoF) information of the grasps.

isaac_sim_grasping/managers.py

Lines 21 to 46 in f3a0685

	def __init__(self, grasps_path, grippers_path, objects_path, controller= 'default',
	world=None):
	# Loading files and urdfs
	self.world = world
	print("Loading File: ", grasps_path)
	with open(grasps_path) as fd:
	self.json = json.load(fd)
	self.gripper = self.json['gripper']
	self.object = self.json['object_id']
	# Extract grasps
	self.grasps = self.json['pose']
	self.dofs = self.json['dofs']
	self.dofs = np.asarray(self.dofs) #graspit_dofs
	self.grasps = np.asarray(self.grasps) #graspit_pose
	# Check for usds Object's and Gripper's
	self._check_gripper_usd(grippers_path)
	self._check_object_usd(objects_path)
	# Verbosity of json data loading
	self.n_jobs = self.grasps.shape[0]
	print("Number of Grasps: " + str(self.n_jobs))
	# GRIPPER SPECIFIC DATA
	self._init_gripper_dicts()

Within the managers.py you can find the hyperparameters used for filtering the grasps of each gripper. In order to reproduce the paper results, one must run the filter with the same hyperparameters:

isaac_sim_grasping/managers.py

Lines 67 to 153 in f3a0685

	def _init_gripper_dicts(self):
	""" GRIPPER INFORMATION INITIALIZATION
	Every gripper should have its information added here
	"""
	#End effector axis (+/- 1,2,3) x,y, z respectively
	self.EF_axes = {
	"fetch_gripper": 1,
	"franka_panda": 3,
	"sawyer": 3,
	"wsg_50": 3,
	"Barrett": 3,
	"robotiq_3finger": 2,
	"jaco_robot": -1,
	"Allegro": 1,
	"shadow_hand": -2,
	"HumanHand": -2,
	"h5_hand": -3
	}
	#Custom Physics dts (increase filtering speed)
	self.dts = {
	"fetch_gripper": 1/120,
	"franka_panda": 1/120,
	"sawyer": 1/120,
	"wsg_50": 1/120,
	"Barrett": 1/120,
	"robotiq_3finger": 1/120,
	"jaco_robot": 1/120,
	"Allegro": 1/120,
	"shadow_hand": 1/120,
	"HumanHand": 1/120,
	"h5_hand": 1/120
	}
	# Direction for DoFs to close gripper
	self.close_dir= {
	"fetch_gripper" : [1,1],
	"franka_panda": [-1, -1], # NOTE, franka_panda gripper by default is closed, so need to open before, Opendir = [1, 1]
	"sawyer": [1, 1],
	"wsg_50": [1, -1], # NOTE, wsg_50 gripper by default is closed, so need to open before, Opendir = [-1, 1]
	"Barrett": [0, 0, 1, 1, 1, 0, 0, 0],
	"jaco_robot": [1, 1, 1],
	"robotiq_3finger": [0, 0, 1, 1, 1, 1, 1, 1, 0, 0, 0],
	"Allegro": [0, 0.5, 0, 0, 1, 0, 1, 1, 0, 1, 0, 0, 0, 0, 0, 0],
	"HumanHand": [0, 0, 0, 0, 0, 1, 1, 1, 1, 0.75, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
	"shadow_hand": [0, 0, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 1, 0, 0, 1, 0, 0, 1, 0, 1],
	"h5_hand": [1,-1,0,0]
	}
	#List of names of joints to check for collisions; it must be as specified in the .usd of the gripper
	self.contact_names= {
	"fetch_gripper" : ["l_gripper_finger_link_joint","r_gripper_finger_link_joint"],
	"franka_panda": ["panda_hand", "panda_leftfinger", "panda_rightfinger"],
	"sawyer": [ "leftfinger", "rightfinger"],
	"wsg_50": ["gripper_left", "gripper_right"],
	"Barrett": ["a_link2_joint","a_link3_joint","b_link2_joint","b_link3_joint","c_link2_joint_0","c_link3_joint"],
	"jaco_robot": ["jaco_8_finger_index", "jaco_8_finger_thumb", "jaco_8_finger_pinkie"],
	"robotiq_3finger": ["RIQ_link_1_joint_a", "RIQ_link_1_joint_b", "RIQ_link_1_joint_c",
	"RIQ_link_2_joint_a", "RIQ_link_2_joint_b", "RIQ_link_2_joint_c",
	"RIQ_link_3_joint_a", "RIQ_link_3_joint_b", "RIQ_link_3_joint_c"],
	"Allegro": [ "link_0_0", "link_2_0","link_3_0", "link_4_0", "link_5_0","link_6_0","link_7_0","link_8_0","link_9_0","link_10_0",
	"link_11_0","link_12_0","link_13_0","link_14_0","link_15_0"],
	"HumanHand": ["index1_joint","index2_joint","index3_joint", "mid1_joint","mid2_joint","mid3_joint", "pinky1_joint","pinky2_joint","pinky3_joint",
	"ring1_joint","ring2_joint","ring3_joint", "thumb1_joint","thumb2_joint","thumb3_joint" ],
	"shadow_hand": [ "index_finger_proximal", "index_finger_middle", "index_finger_distal",
	"little_finger_proximal", "little_finger_middle", "little_finger_distal",
	"middle_finger_proximal", "middle_finger_middle", "middle_finger_distal",
	"ring_finger_proximal", "ring_finger_middle", "ring_finger_distal",
	"thumb_proximal", "thumb_middle", "thumb_distal"],
	"h5_hand" : ["left_tip_link", "right_tip_link"]
	}
	#Amount of contacts required for the grasp to be considered as ready
	self.contact_ths = {
	"fetch_gripper" : 1,
	"franka_panda": 1,
	"sawyer": 1,
	"wsg_50": 1,
	"Barrett": 1,
	"jaco_robot": 1,
	"robotiq_3finger": 1,
	"Allegro": 1,
	"HumanHand": 1,
	"shadow_hand": 1,
	"h5_hand": 1
	}

All the files provided within this repository and in the [dataset folder] are set up for use out of the box. To reproduce the results you will need to run the simulation on the dataset and save to a different folder (output_dir). This will override the fall_time values and save the latest results on the new file with the relevant information: grasp pose information, grasp dof information, test type, total test time, fall time. Below is the corresponding manager.save_json function:

isaac_sim_grasping/managers.py

Lines 246 to 277 in a7207a1

	def save_json(self,output_path):
	""" Saves json on disk.
	Args:
	out_path: path to save json at
	"""
	print("Saving File at: ",output_path)
	new_json = self.json
	#Single elements
	new_json['standalone'] = "standalone"
	new_json['gripper'] = self.gripper
	new_json['object_id'] = self.object
	new_json["test_type"] = self.test_type
	new_json['test_duration'] = self.total_test_time
	tmp =np.logical_and(self.slip_time==-1,self.fall_time!=-1)
	self.slip_time[tmp] = self.total_test_time #Didn't even slip
	#Lists
	new_json['pose'] = self.grasps.tolist()
	new_json['dofs'] = self.dofs.tolist()
	new_json["fall_time"] = self.fall_time.tolist()
	new_json["slip_time"] = self.slip_time.tolist()
	#NEW
	new_json['runtime'] = time.time()-self.init_time
	new_json['physics_dt'] = self.physics_dt
	new_json['final_dofs'] = self.final_dofs.tolist()
	with open(output_path,'w') as outfile:
	json.dump(new_json,outfile)
	return

Note: To reproduce the transferred grasp results the same must be done with the transfer manager class T_manager.

More Documentation

• Adding a new Gripper
• Adding a new Object
• Available Gripper Controllers
• Graspit Grasp Generation Code

Helpful Links

• MultiGripperGrasp Shared Folder here
• Google Scanned Objects and YCB objects .usd files here
• Filtered GraspIt Grasps Dataset here
• Transferred Grasps Dataset here
• Isaac Sim Manual: https://docs.omniverse.nvidia.com/isaacsim/latest/overview.html
• Helpful code for python standalones (code snippet samples): https://docs.omniverse.nvidia.com/isaacsim/latest/reference_python_snippets.html
• Installing Packages for use with isaac sim python.sh: https://docs.omniverse.nvidia.com/isaacsim/latest/installation/install_python.html

Notes

• Deactivate conda if you have an active environment, it may cause some errors when running isaac sim.
• Always use complete paths for the directories, errors may occur otherwise.