A multimodal classifier with a force-text pair from the interaction between the robotic arm and granular materials (GMs).
This repository contains the source code and dataset GM10-ts and GM10-ts-Plus of the paper "A Joint Learning of Force Feedback of Robotic Manipulation and Textual Cues for Granular Material Classification", which is accepted by RA-L (2025).
A comprehensive dataset GM10-ts-Plus (10 GMs, 27,000 data points) has been uploaded.
- Ubuntu 20.04 or Windows 11
- Python 3.9.15 (or above)
- Pytorch 1.13.1
- cuda 11.7
- CLIP 1.0
In each data point (CSV file) in the dataset, each row refers to a raking experiment.
If defining each row has M columns, then the 1-st column refers to the GM id.
- In GM10-ts:
From the 2-nd column to the (1+(M-1)/2)-th column, they are time series.
From the (2+(M-1)/2)-th column to the end, they are force series.
- In GM10-ts-Plus:
From the 2-nd column to the 4-th column, they are (a, v, d) values.
From the 5-th column to the (5+(M-4)/2)-th column, they are time series.
From the (6+(M-4)/2)-th column to the end, they are force series.
Any problem, feedback, issue, or bug-finding is welcome as an open-source library.
Contact: Benji Z. Zhang (zzqing@connect.hku.hk)