Franka vision grasping

This repo is for grasping task on the following setup.

Arm: Franka Emika Panda

Vision Sensor: Intel RealSense D435

Computing Platform: Jetson AGX Xavier

Requirements

• Jetpack 4.4 with Realtime Kernel
• Set user task priority for realtime kernel
• libfranka version == 0.7.1
• opencv with contrib (for calibration with aruco marker)
• jetson-inference
• librealsense2
• pyrealsense2 (build from source)
• pybind11
• Eigen3
• Python3
  • numpy
  • yaml
  • matplotlib (for calibration visualization)
  • pytransform3d (for calibration visualization)

TODO

• complete README (currently the usage info is not fully correct and it requires additional setup related with Jetson-inference)
• Add yolov5 example

The requirement info is correct

Usage

DetectNet trained on COCO-bottle

python3 vision_grasping.py --network=coco-bottle

Self-trained ONNX model

python3 vision_grasping.py --model=<path-to-your-model> --labels=<path-to-your-label> --input-blob=input_0 --output-cvg=scores --output-bbox=boxes

Troubleshooting

• After flash the Jetson Xavier with JetPack 4.4 and realtime kernel patch, sudo jetson_clocks may not able to turn on the fan, which will lead to severe heat accumulation. You can use echo 255 | sudo tee /sys/devices/pwm-fan/target-pwm to manually set the fan at maximum speed.

  Reference: https://forums.developer.nvidia.com/t/fan-management-xavier/70166/9