9DTact

Table of contents

1. Overview
2. Installation
3. 3D Shape Reconstruction
   1. Camera Calibration
   2. Sensor Calibration
   3. Shape Reconstruction
4. 6D Force Estimation
   1. BOTA Driver
   2. Data Collection
   3. Data Processing
   4. Model Training
   5. Force Estimation
5. Run in ROS
   1. Shape Reconstruction in ROS
   2. Force Estimation in ROS
   3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS

Overview

This repository contains the code and the hardware source files for the paper:

9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation
Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu
RAL, 2023
Website / Arxiv Paper / Video Tutorial / Bom (CN)

Installation

Create a conda environment:

conda create -n 9dtact python=3.8

Install pytorch (choose the version that is compatible with your computer):

conda activate 9dtact
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

In this repository, install the other requirements:

pip install -e .

3D Shape Reconstruction

For all the terminals used in this section, they are located in the shape_reconstruction directory and based on the 9dtact conda environment:

cd shape_reconstruction
conda activate 9dtact

1. Camera Calibration

Before you start, 3d print the calibration board.
Run:

python _1_Camera_Calibration.py

Just follow the printed instructions.

2. Sensor Calibration

Firstly, prepare a ball with a radius of 4.0 mm. (The radius of the ball depends on the thickness of the sensor surface. 4.0 mm is just a recommendation.)
Then, run:

python _2_Sensor_Calibration.py

Just follow the printed instructions.

3. Shape Reconstruction

python _3_Shape_Reconstruction.py

6D Force Estimation

1. BOTA Driver

If you want to collect force data with a BOTA MiniONE Pro sensor, you need to:
Create a directory named 'bota_ws' as the ROS workspace, and install the bota driver package.

2. Data Collection

At the first terminal, open the BOTA sensor:

cd ~/xxx/bota_ws # Modify 'xxx' to enter the workspace directory
source devel/setup.bash
roslaunch rokubimini_serial rokubimini_serial.launch

At the second terminal, run:

source ~/xxx/bota_ws/devel/setup.bash
cd data_collection
conda activate 9dtact
python collect_data.py

At the third terminal, open the 9DTact sensor:

cd shape-force-ros
conda activate 9dtact
python _1_Sensor_ros.py

3. Data Processing

Open a terminal, normalize the wrench:

cd data_collection
conda activate 9dtact
python wrench_normalization.py  # remember to modify the object_num

At the same terminal, split the data by running:

python split_train_test.py

and also:

python split_train_test(objects).py

4. Model Training

To train the model on the stadard training dataset, run:

cd force_estimation
python train.py --model_name="Densenet" --model_layer=169 --optimizer="ADAM" --lrs=False --image_type="RGB" --cuda_index=6 --resize_img=False --train_mode=True --test_object=False --mixed_image=True --pretrained=False --batch_size=64 --num_epoch=200 --learning_rate=5.0e-4 --weight_decay=0.0

You may also choose to use Weights and Bias (wandb) by setting use_wandb as True, which helps to track the training performance.

5. Force Estimation

You need to specify a model saved in the 'saved_models' directory as an estimator, by modifying the 'weights' parameters in the force_config.yaml.
After that, run:

cd force_estimation
python _1_Force_Estimation.py

Run in ROS

1. Shape Reconstruction in ROS

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _2_Shape_Reconstruction_ros.py

2. Force Estimation in ROS

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py

(Optional for visualization) At the third terminal, open the visualization window:

cd force_estimation
conda activate 9dtact
python force_visualizer.py

3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS

At the first terminal, open the force estimator:

cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _4_Shape_Force_ros.py